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ABSTRACT 

Academy of READING (AoR) is designed to improve students’ foundational reading 
skills; it is used in almost all Wake County Public Schools (WCPSS) secondary schools. 
Cen tral staff recommended students with prior End of Grade/Course (EOG/EOC) scale 
scores placing them high in Level II or low in Level III have first priority for service; 
many students sensed scored below this range (second priority). Program completion 
rates increased from 2008-09 (26%) to 2009-10 (41%), but were still considerably 
lower than desired (100%). Growth for 2009-10 AoR participan ts on Enel of Grade 
reading/English tests was strong (with the percentage of students reaching their growth 
targets increasing from 37.5% before service to 60.1% afterwards). This improvement 
is educationally important. Program completion gave students only a slight advantage 
in terms ofEOG growth. The program appeared to be helpfid, with patterns generally 
favoring middle school over high school students. 
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SUMMARY 


Students Served 

Use of AoR software has expanded across schools over the last several years in WCPSS, with 55 
of 58 secondary schools (95%) utilizing the program by 2009-10. The AoR software program 
served 3,640 students in 2009-10 — much higher than the 1,358 students served in 2008-09. 
Schools appeared to use many different sets of criteria to identify students to receive service in 
AoR. In 2008-09 and 2009-10, fewer students scored in the primary EOG range (of high Level 
IIs and low Level Ills) recommended by central staff than scored below it (the secondary target 
range). Assignment is a multi-step process, and the AoR teachers reported using a number of 
criteria to determine students to be served, including EOG/EOC scores, other assessments, 
Individual Education Plans, ESL status, and teacher recommendations. 

Implementation 

AoR was most commonly utilized in remedial electives in 2009-10 (44%), with Curriculum 
Assistance and regular English I classes the next most common (at about 15% each). Students 
work on computer-based lessons related to foundational reading skills with teacher support as 
needed. Overall, the program appeared to be well implemented. Contracted AoR coaches from 
the vendor (TE21) found most teachers were implementing all key elements of the program, and 
97% of survey respondents reported receiving training. 

Aspects of the program’s use of technology, the content, the pacing, the motivational aspects, 
and the data provided were mentioned both as likes and disl ik es. 

• Teachers appreciated the focus on basic skills, the self-pacing, the motivators, the available 
data feedback, and the opportunity to work one-on-one with students. 

• On the other hand, a number of teachers indicated that the high standards for accuracy and 
rapid responses (automaticity) for some exercises led to student frustration and/or boredom. 
Teachers also reported issues related to technology glitches, slow pacing, immature 
characters used in the materials, and difficulty in understanding the narrator. A common 
request was also that some vocabulary and more comprehension be added to the program. A 
few schools had issues of computer access and dependability. 

Outcomes 

After completing a pretest, students are assigned a training stream in AoR. The length and 
number of skills each student must master varies depending on their performance on the pre- 
test). The objective was that all students would complete at least 80% of their program ( a very 
high standard). This objective was not met either year, with AoR program completion rates at 
only 26% in 2008-09 and 41% in 2009-10. The percentage of students completing the program 
between 2008-09 and 2009-10 did increase; this could relate to better implementation at schools 
with the program for two or more years or the fact that more middle schools were involved in 
2009-10 than 2008-09. 
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The EOG/EOC program objective was that students would improve in their reading 
comprehension as measured by Reading or English EOG/EOC; this objective was met. Based on 
the 2009-10 cohort, the percentage of students reaching or exceeding their growth target 
increased from 37.5% before service to 60.1% after service. Comparisons of average academic 
growth scores for Spring 2009 and Spring 2010 also indicated statistically significant 
improvement. Analyses revealed a medium effect size, which is considered educationally 
significant. Students appeared to benefit, on the average, whether they finished the program or 
not (with a slight advantage for those completing at least 80% of their training program). 

Cost Benefit 

While AoR has been a considerable investment for WCPSS over time ($1.8 million), the 
continuing costs are considerably lower than the initial costs of site licenses and training. If the 
cost of the licenses is amortized over time, and annual costs are added, the cost per student in 
2009-10 was $148. The cost per student will vary by school depending on the number of 
students served per school; the cost per school averages out to $9,779. The cost per successful 
student (based on those with academic change scores at or above 0) was $246; those with costs 
below this average included students who scored high in Level II and in the low to mid-range of 
Level III. However, the number of students who improved was far greater in the high Level I 
and middle Level II range, with a slightly higher cost per successful student of $253 to $262. 

Recommendations 

Overall, AoR did appear to meet the need of improving many students’ foundational reading 
skills. Continuation of AoR is recommended, with the following areas for improvement: 

• Target services to student needs more strategically, 

• Improve the program to increase student success, including targeted training for teachers to 
clarify some misconceptions, 

• Consider the best classroom structure and level for these services. 
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ACADEMY OF READING® (AoR) 2008-09 and 2009-10 EVALUATION 


This bulletin is organized around key questions of interest. AoR implementation and one-year 
impact are examined. 

Question: What Need Does AoR Address? What Are Its Goals? 

The AoR is a reading intervention software tool designed to complement an existing reading 
curriculum for students who have gaps in their foundational 
reading skills. The program is designed to build accuracy and 
automaticity in sound matching, letter-sound matching, decoding, 
phonics, and fluency to prepare students to develop 
comprehension skills. Because reading is so important to success 
in many high school subjects, improved reading skills should lead 
to increased reading comprehension and success in high school. 

The logic model shown on page 2 was developed collaboratively 
between the Evaluation and Research Department (E&R) staff 
and program staff to illustrate the need, short-term, intermediate, 
and long-term goals of the AoR. Briefly, program goals are to: 

• Help students master the skills in the AoR software the 
semester or year they are served, 

• Improve EOG/EOC growth in reading and English the year of service and over time, 

• Increase the percentage of grades of C or better for courses over time, and 

• Ultimately increase graduation rates in the long-term. 

The WCPSS literacy team recognizes the AoR as one option available to struggling readers in 
middle and high schools. Being able to read fluently is considered essential to high school 
success. Other remedial approaches may be better suited for students with greater deficits in 
reading, which involve more teacher-directed, multi-sensory reading instruction. Likewise, 
students who possess adequate foundational reading skills but struggle with issues such as 
comprehension or vocabulary may be better served by instruction and resources designed with 
that focus. Regardless of the approach or resource, teachers must use diagnostic and formative 
assessments to determine how best to tailor instruction to meet students’ needs. Professional 
development and resources for this continuum of instructional needs are made available to 
schools. 


“Automaticity. . . is defined 
by the program as learning 
skills to a level of rapid 
automatic responding 
through practice. ” 

Florida Center for Reading 
Research (2004) 


5 


Academy of Reading Evaluation 


E&R Report No. 10.23 


Figure 1 

AoR Logic Model for WCPSS 



Curriculum and Instruction (C&I) contacts selected Academy of READING (AoR) based on 
research cited by Rissman (2004). One of these studies, by Fiedorowicz (1986), was a small 
experimental study with 16 students; he found that student participants receiving the program 
performed significantly better on standardized measures of reading accuracy and speed than 
those who did not. However, the two groups were not significantly different on an audio-visual 
matching task. A larger quasi-experimental study of 1 15 reading disabled students from five 
districts in Ontario (Fiedorowicz and Trites, 1987) also found that participants who received the 
AoR training showed significant improvement on standardized measures of word recognition 
measures and speed and accuracy of paragraph reading compared with matched students who did 
not. A more recent study by Foh (2009) studied results for 1,945 high school students across 13 
states who had participated in AoR. He found gains averaging 1.6 grade levels on program 
measures for high school students, with the greatest gains for those with 10-20 hours of time on 
task. Most students can accomplish this in one semester, although some may need one year. 
AutoSkill (2007) also indicates that the AoR product is responsive to small experimental studies 
that found improved brain functioning (based on pre- and post-imaging) and reading skills for 
dyslexic readers who were given a three- week, phonologically driven instructional treatment. 
Beyond these studies, the Best Evidence Encyclopedia reviewed a variety of reading programs 
for secondary students; they found no studies that met their criteria for experimental and control 
studies on AoR, and therefore did not rate the effectiveness of AoR (Johns Hopkins University 
School of Education's Center for Data-Driven Reform in Education [CDDRE], 2008). 
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Question: What Are the Evaluation Questions of Interest? 

The E&R program accountability staff met with the Director for K-12 literacy and the Senior 
Directors for Middle and High Schools to discuss goals for the program and ways to measure its 
effectiveness. E&R staff then drafted the evaluation plan, with the key evaluation questions and 
data sources shown in Table 1. Program staff played a critical role in providing data for many of 
the questions. Subsequent meetings and communications have been primarily with the Director 
of Literacy to obtain necessary data and additional details and clarifications. 

Table 1 

Key Evaluation Questions and Data Sources — AoR 2008-09 and 2009-10 


Question 

Data Source 

1. What need is the program designed to 
address? What are its goals? 

Program staff interviews 
(turned into logic model) 
2009-10 teacher survey 

2. What are the key strategies used in AoR? 

Program staff and vendor (TE21) 

Websites 

Articles 

2009-10 teacher survey 

3. Which schools have utilized AoR and for 
how long? 

Program staff 

4. Which students were targeted and actually 
served by the program in 2008-09 and 
2009-10? 

Program staff 
Participant files 
EOG/EOC test records 
2009-10 teacher survey 

5. Was the program implemented with fidelity 
in 2009-10? 

Trainer site visit reports 

6. Did the program meet its short-term goals 
for program completion and for EOG/EOC 
growth? 

Participant files 
EOG/EOC test records 

7. What costs and resources are needed to 
implement AoR? What benefits are 
anticipated? Was the program cost- 
effective in 2009-10? 

Program staff — budget records and 
expectations 

Studies on program effectiveness and cost 
benefit 

Cost effectiveness initial analysis 

8. Should the program be continued as is, 
modified, or discontinued? 

All (see Recommendations) 


Information on methodology and analyses is included in Appendix 1. Intermediate and long- 
term benefits cannot be addressed at this time. In a future year, we hope to have the opportunity 
to analyze the 2009-10 student cohort over time in terms of future grades earned, EOG growth, 
and dropout rates. 
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Limitations of this Study 

Students who did not complete at least 80% of the program could not be included in the analysis 
of improvement across the AoR program categories unless they took the post-test; we must 
assume that these students likely improved less in their foundational reading skills than other 
students. Therefore, readers must be cautious about overstating the effectiveness of AoR in 
terms of improvement within AoR. Non-completing students were included in the EOG/EOC 
analyses of growth (if they had appropriate EOG/EOC scores), so those analyses were not 
impacted. 

Prior EOG/EOC Reading growth test scores for Spring 2008 and/or Spring 2009 were missing 
for about 20% of the students in this study. Missing test history data prohibits us from 
determining whether students were appropriate for the program or made achievement growth 
over time. These students may have been new to the district in 2008-09, not present during 
testing, exempt from testing as new Limited English Proficiency (LEP) students in their first year 
of school in the U.S., or not in the appropriate high school course that year to be tested. In 
addition, Students with Disabilities (SWD) who took alternate forms of the EOG and EOC 
would not have scale scores, which are the basis for growth analyses. 

Cost benefit analyses can be approached in a variety of ways. Different methods could lead to 
somewhat different results. We utilized cost effectiveness ratios based on the percentage of 
students meeting their expected academic change score. Cost benefit results should always be 
considered in the context of other results in reaching conclusions about projects. 

The teacher survey had a low return rate at 35%. Still, the 63 responses seemed representative 
enough to inform our understanding of WCPSS program implementation. Middle school 
teachers represented 80% of respondents, slightly higher than the middle school students in AoR 
(67%). Multiple choice and open-ended responses provided insights about the variability in 
selection methods, implementation issues, and teacher misunderstandings about AoR. As 
always, readers must just be cautious in the types of generalizations made from the survey data. 

Lastly, a comparison group was not feasible to establish for this study. As a post hoc analysis, 
we could match students based on prior EOG/EOC scores, but we did not have scores available 
to match students on a measure of foundational reading skills. Also, some students in AoR may 
have also been in Integrated Reading (IR), which is sometimes combined with AoR. Since 
students were in various class configurations, we must assume that AoR would still have a 
discernable impact even without other supports. The widespread improvement across schools 
sharing the common element of participation in the AoR strengthens our assumption that the 
program made a difference for students. If a future study is launched, we will explore the 
feasibility of establishing a comparison group again. 


Question: What Are the Key Strategies Used in AoR? 

AoR focuses on the foundational reading skills of sound matching, letter-sound matching, 
decoding, phonics, and fluency. AoR includes only a limited amount of work on comprehension 
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at the end of the training sequence and no material of vocabulary. Improving foundational s ki lls 
is considered essential to helping students increase comprehension. Teachers of AoR in 2009-10 
were asked how helpful they felt AoR was in addressing various reading skills areas. Their 
perception matched the program’s coverage fairly well. Most teachers realized that AoR was 
most helpful for phonemic awareness, phonics, and fluency. Most also understood that AoR was 
less helpful with comprehension and vocabulary. However, 30% of the respondents indicated 
that AoR was somewhat or very helpful with comprehension, and 40% indicated it was 
somewhat or very helpful with vocabulary. This represents a misunderstanding of AoR. A 
number of teachers requested better coverage of comprehension and vocabulary as possible 
improvements to the product. 

In WCPSS, AoR is used for students within some regular English/Language Arts classes, within 
electives, within Curriculum Assistance (CA) courses, and in English as a Second Language 
(ESL) classes. Students utilize the software program to focus on their foundational reading skills 
three to five times per week for 30-40 minutes per session over a period of one to two semesters. 
The program measures the students’ actual time spent on task, with a goal of at least 45 minutes 
of time spent on task per week. Time on task does not include time spent on activities such as 
start-up, shut-down, taking assessments, visiting the Trophy Room, or changing avatars. Loh 
(2009) studied results for 1,945 high school students across 13 states. He found gains averaging 
1 .6 grade levels on program measures for high school students, with the greatest gains for those 
with 10-20 hours of time on task. (In WCPSS, most students could have 10-20 hours of time on 
task during one semester of service.) In addition, Loh reported that 40% of the students gained 
two grade levels (categories) or more after participating in the program. 

Students’ starting level is based on a screening test that consists of 
several maze passages. Students are then assigned a “training stream” 
and move through an individually assigned series of tutorials, activities, 
and assessments with the goal of mastering 80-100% of the material on 
each skill assigned. Students who start at the lowest levels (Below 
Basic and Basic) are considered below grade level and have to 
complete more material to complete their training stream than those 
who score Proficient, Advanced, or Graduate. Once students have 
completed their training stream, they take the post-assessment, again 
consisting of several maze passages. 

Teachers serve actively as coaches while students are working on their tasks. Teachers have 
constant access to a monitoring system on their own computer which allows them to see exactly 
what errors a student is making and constantly updates to list students in priority order of need 
for teacher coaching. In order to support teachers in learning how to best coach students, 
coaching strategies, training, observations, review, and support are provided by the AoR training 
and support vendor (a company named TE21). In addition, each school has its own Site 
Coordinator who works with the District Coordinator and Technology Services to schedule 
training, troubleshoot technological issues, and encourage best practice. 


r *\ 

Maze passages are 
assessments in which 
students choose the most 
appropriate word from 
several options to 
complete a passage 
within a time limit (e.g., 
three minutes). 

< J 


9 


Academy of Reading Evaluation 


E&R Report No. 10.23 


Question: Which Schools Have Utilized AoR? 

AoR was rolled out to schools over time in WCPSS. Table 2 illustrates schools with the program 
each year. Three schools had the program before 2007-08. Service was rolled out first to high 
schools, because staff believed there were fewer existing resources available to help struggling 
readers compared to middle school, and that reading skills were critical to success in high school. 
In 2009-10, many middle schools were added. Presently, nearly all secondary schools in 
WCPSS (55 of 58) have this resource — 33 middle schools and 22 high schools. 

Table 2 

Years of Service in AoR by School 


No. 

Middle 

Schools 

2007-08 
or Before 

2008- 

09 

2009- 

10 

High Schools 

2007-08 
or Before 

2008- 

09 

2009- 

10 

1 

Carnage 

X 

X 

X 

Millbrook 

X 

X 

X 

2 

Heritage 

X 

X 

X 

Fuquay- Varina 

X 

X 

X 

3 

Moore Square 

X 

X 

X 

Mary Phillips 

X 

X 

X 

4 

East Garner 


X 

X 

Apex 


X 

X 

5 

Fuquay 


X 

X 

Athens Drive 


X 

X 

6 

Holly Ridge 


X 

X 

Broughton 


X 

X 

7 

Reedy Creek 


X 

X 

Cary 


X 

X 

8 

Mt. Vernon 


X 

X 

East Wake 


X 

X 

9 

Wake Forest 


X 

X 

Enloe 


X 

X 

10 

Zebulon 


X 

X 

Green Hope 


X 

X 

11 

Longview MS 


X 

X 

Knightdale 


X 

X 

12 

Apex 



X 

Longview HS 


X 

X 

13 

Carroll 



X 

Leesville Road 


X 

X 

14 

Centennial 



X 

Middle Creek 


X 

X 

15 

Daniels 



X 

Panther Creek 


X 

X 

16 

Davis Drive 



X 

Sanderson 


X 

X 

17 

Dillard 



X 

Southeast Raleigh 


X 

X 

18 

Durant Road 



X 

W ake Early College 


X 


19 

East Cary 



X 

Wake Forest-Roles ville 


X 

X 

20 

East Wake 



X 

W akefield 


X 

X 

21 

East Millbrook 



X 

Garner 



X 

22 

Leesville Rd 



X 

Holly Springs 



X 

23 

Ligon 



X 





24 

Lufkin Road 



X 





25 

Martin 



X 





26 

North Garner 



X 





27 

River Oaks 



X 





28 

Salem 



X 





29 

Wakefield 



X 





30 

Wendell 



X 





31 

West Cary 



X 





32 

West Lake 



X 





33 

West Millbrook 



X 






Note 1: Moore Square MS had no students enrolled in 2008-09; Green Hope HS had none enrolled in 2009-10. 
Note 2: Holly Grove MS started in 2010-11; Mills Park MS and Heritage HS anticipate starting in 2010-11. 
Note 3: Longview, River Oaks, and Mt. Vernon MS, Mary Phillips and Longview HS are alternative schools. 
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Question: Which Students Were Targeted and Actually Served by AoR? 

Students Targeted 

WCPSS staff considers AoR most appropriate for those students with some gaps, but not 
extreme gaps, in foundational skills. Central staff recommends students in the top 25% of Level 
II or the bottom 25% of Level III as most likely to benefit from the program (first priority). Staff 
emphasizes that AoR not be the sole resource used for students with the most foundational needs 
(those in Level I primarily) because it might not be intensive or multi-modal enough on its own. 
Central staff did not change this recommendation when EOG cut scores were raised in 2007-08. 
In 2008-09 and 2009-10, the targeted group for AoR was to be based primarily on the pattern of 
EOG scores over the past two years. District staff recommended that: 

• Students who had a pattern of EOG scale scores in the top 25% of Level II and the bottom 
25% of Level III in reading or English I for the past two or three years were considered those 
most likely to make gains through AoR. Students with declining scores (e.g., Level III to 
Level II) were also possible candidates. These students were considered first priority for 
service. Beyond the first priority group, if space was available, students in Level I or the 
bottom 75% of Level II were allowed into the program as the next most likely to make gains. 

• The resource would be used primarily with students in grades 6-9, with some older students 
who had been retained at some point or still had gaps in reading skills. 

• In addition, students on an Occupational Course of Study (OCS) could also be considered, if 
AoR was in addition to direct reading instruction provided by a teacher. 

Authorized AoR trainers from TE21 sometimes expressed the opinion that the resource was 
helpful to a broader range of students — Level I and Level II. 

WCPSS central contacts ask that teachers review these data for the past two years in making 
their recommendation. This is intended to be a multi-step process: 

• The sending teachers make a recommendation that the student receive additional support in 
Reading/Language Arts or a remedial elective the next year. These recommendations are 
made before the state tests are given for the current year, so they reflect the prior year’s test 
scores. Teachers also use other data in recommending students likely to benefit from AoR — 
including screening assessment scores, classroom performance, and teacher or SST 
recommendations of students. 

• The specific student placement is decided by the receiving school staff (e.g., administrators 
and master schedulers) who schedule students for the year. These schedulers should check 
the most recent EOG/EOC score to confirm the recommendations. Students are typically 
placed in a remedial or regular-level class by the scheduler. Some regular English courses 
may include primarily students with lower scores the prior year. 
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• AoR is considered a supplemental tool by the receiving teacher. 

Teachers screen the students most likely to need the resource 
with one or more short assessments made available by the 
Consortium on Reading Excellence (CORE) and then make their 
decision about who is served in AoR (Diamond, 2008). Once 
students are selected, they take the AutoTest Cloze Paragraph 
Assessment to determine their starting place and training stream 
in the program. 

Students Served 

As shown in Table 3, the AoR served 1,358 students in 2008-09. In 
terms of student characteristics, FRL, SWD, LEP, Black/ African 
American, and Hispanic/Latino students are over-represented 
compared to WCPSS characteristics overall. White students are 
under-represented. This pattern of service is appropriate in that the 
subgroups that are over-represented are more likely to score below grade level on the EOG tests. 
The most striking difference is for SWD students (44% of students using AoR vs. 14% in 
WCPSS). Females are also underrepresented relative to males. 

In 2009-10, the number of students served in AoR more than doubled compared to 2008-09, 
from 1,358 to 3,640 (as the number of campuses increased). The profile of students served was 
similar in 2009-10 and 2008-09 by gender and racial/ethnic group, but some differences of note 
were found in FRL and SWD percentages. The percentage of students who had low income (as 
indicated by FRL status) increased from 50% to 63%, and the percentage of SWD decreased 
from 44% to 30%. 


CORE assessments are 
provided by the 
Consortium on 
Reading Excellence 
( CORE) as quick 
screening tools. 

Cloze assessments 
measure understanding 
of words in context. 

Students fill in blanks 
in a paragraph or 
passage with no 
options provided 
(within a time limit). 
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Table 3 

Student Characteristics, 2008-09 and 2009-10, Grades 6-12 


Characteristic 

AoR Students 

WCPSS Middle and High 

Year 

2008-09 

2009-10 

2008-09 

2009-10 


Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

FRL 

684 

50% 

2,280 

63% 

17,770 

23% 

20,763 

29% 

SWD 

602 

44% 

1,087 

30% 

9,536 

14% 

9,630 

13% 

LEP 

294 

22% 

682 

19% 

4,408 

6% 

4,214 

6% 

Female 

565 

42% 

1,652 

45% 

35,787 

51% 

35,176 

49% 

Male 

793 

58% 

1,988 

55% 

34,451 

49% 

36,675 

51% 

American Indian 

4 

0 

12 

0% 

202 

0% 

208 

0% 

Asian 

41 

3% 

129 

4% 

3,827 

5% 

4,075 

6% 

Black/African Am. 

638 

47% 

1,819 

50% 

19,875 

28% 

20,124 

28% 

Hispanic/Latino 

290 

21% 

752 

21% 

6,621 

9% 

7,123 

10% 

Multiracial 

55 

4% 

167 

5% 

2,631 

4% 

2,917 

4% 

White 

330 

24% 

761 

21% 

37,082 

53% 

37,404 

52% 

Total 

1,358 


3,640 


70,238 


71,851 



Note: 1: Students may appear in more than one category: race and gender, FRL, SWD, and/or LEP. 

2: Participants total 1,358 for 2008-09 and 3,640 for 2009-10 within race and gender categories. 
Data Source: 2008-09 and 2009-10 WCPSS Annual Demographic Report - www.wcpss.net/demographics 


We also explored the grade level distribution of those served and the assessments taken based on 
the 2008-09 and 2009-10 cohorts. Results shown in Tables 4 and 5 suggest that: 

• Middle school use exceeded high school use in both 2008-09 and 2009-10. Students in 
grades 6 used the program most heavily both years. At the high school level, grade 9 had the 
heaviest use. 

• Over three-quarters of participants took the regular EOG/EOC in the year prior to 
participation, with about 10% in the 2008-09 cohort and 7% in the 2009-10 cohort taking 
alternate assessments such as North Carolina Checklist of Academic Standards (NCCLAS), 
NCExtendl, or NCExtend 2. Over 10% had no scores available for both cohorts. 


13 





Academy of Reading Evaluation 


E&R Report No. 10.23 


Table 4 

2008-09 AoR Participants and Tests Taken in 2007-08 


2007-08 Test Type 

Grade in 2008-09 


6 

7 

8 

9 

10 

11 

12 

Total 

% 

EOG 

289 

193 

179 

276 




937 

69.0% 

EOC 




48 

45 

2 

1 

96 

7.1% 

NCCLAS 




5 

3 



8 

0.6% 

NCEXTEND1 

2 

2 

4 

2 




10 

0.7% 

NCEXTEND2 

34 

24 

25 

17 




100 

7.4% 

NCEXTEND2-OCS Eng I 





9 

1 


10 

0.7% 

None 

24 

25 

24 

70 

14 

32 

8 

197 

14.5% 

Total 

349 

244 

232 

418 

71 

35 

9 

1,358 

100.0% 


Note: All tests are displayed above, regardless of score or exemption. Nine students with blank scores are included 
in the above counts under the corresponding test. 

Program staff indicated that students served in grades 1 1 and 12 would be more likely to be 
Occupational Course of Study (OCS) students. These students do take the Extend II assessments 
for English I, Algebra I, and Life Skills courses. While 10 students are shown taking OCS for 
English I, additional OCS students could be in the “None” category if they completed the 
English I test before grade 11. 


Table 5 

2009-10 AoR Participants and Tests Taken in 2008-09 


2008-09 Test Type 

Grade in 2009-10 


6 

7 

8 

9 

10 

11 

12 

Total 

% 

EOG 

931 

758 

823 

432 




2,944 

80.9% 

EOC 




33 

35 

5 


73 

2.0% 

NCCLAS 

3 

1 


2 




6 

0.2% 

NCCLAS-Eng I 





7 



7 

0.2% 

NCEXTEND1 

2 

3 

6 





11 

0.3% 

NCEXTEND2 

69 

61 

35 

43 




208 

5.7% 

NCEXTEND2-OCS Eng I 





11 

2 

1 

14 

0.4% 

None 

84 

66 

79 

64 

28 

31 

25 

377 

10.4% 

Total 

1,089 

889 

943 

574 

81 

38 

26 

3,640 

1 00% 


Note: All tests are displayed above, regardless of score or exemption. Four students with blank scores are included 
in the above counts under the corresponding test. 

In terms of EOG score patterns, we explored whether students served met the first and second 
priorities for the program in 2008-09 and 2009-10 based on students with EOG/EOC scores 
available at least one year. Figure 2 displays the percentage of students in each score range each 
of the two prior years and combined across years. In Figure 2, we included those meeting the 
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first priority criterion at least one year (rather than both years). When patterns of scores across 
two years are examined for those in AoR in 2008-09, results indicate that about a third (34.4%, 
or 373 of 1,083) of the students scored in the target range, with the largest group (39.3%) having 
scores below the first priority range both years. An additional 21.6% showed a mixed pattern of 
scores that were both outside the target range, with nearly all declining from above to below the 
target range. Based on this loose interpretation of the criteria for first and second priority, 73.7% 
of the students served technically met those criteria. However, second priority students out- 
numbered the first priority students. If we counted only those students who had first priority 
scores both years, only 14 students scored in the target range in both of the two prior years (less 
than 1% of those served). 


Figure 2 

Prior Two Years of EOG/EOC Achievement Ranges for 
2008-09 AoR Students 



39 . 3 % 



2006-07 

2007-08 

2006-08 

■ Above 

388 

46 

50 

■ In 

243 

144 

373 

Mixed 

n/a 

n/a 

234 

Below 

287 

834 

426 


Prior EOG Achievement Ranges 


In 2009-10, the pattern of prior scores for those in AoR was similar to 2007-08 and 2008-09; 
the majority of those served scored below the target range. As shown in Figure 3, when 
combined across years, 33.4% had at least one prior score within the first priority range, with 
61.7% in the second priority range. Thus, the percentage of students scoring in the first priority 
range was about one-third both years, with the percentage below this range increasing in 2009- 
10. Overall, 95% of those served were in the first or second priority range. However, the fact 
that more students were in the second priority than first priority range is an issue. It is not clear 
that all first priority students were considered before enrolling the second priority students. 
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Figure 3 

Prior EOG/EOC Achievement Ranges for 
2009-10 AoR Students 


£ 30 % 


20% 





10 % 


0% 


■ Above 


n/a 


n/a 


2086 


Prior EOG Achievement Ranges 


2008- 09 N= 1,358 

Note: 440 students either took alternate assessments or were missing test scores in 2006-07, and 334 for 2007-08. 
l,083students had at least one test score in 2006-07 and/or 2007-08 (275 were missing both). 

2009- 10 N= 3,640 

Note: 906 students either took alternate assessments or were missing test scores in 2007-08, and 627 for 2008-09. 
3,052 students had at least one test score in 2007-08 and/or 2008-09 (588 were missing both). 


Finally, we reviewed survey results for more information on the criteria that were used to select 
students for AoR. Just over half of the teachers used EOG scores as at least one criterion for 
selecting students. Overall, 29% indicated they used one year of EOG scores; 26% used two 
years. Those who did not use them were fairly evenly split in terms of whether they had access 
to EOG scores or not. Given that the screening and selection process has multiple steps, it is not 
totally surprising that many teachers did not use the EOG scores. Nine of the 63 respondents 
indicated in their comments that the administrative team selected the students, so their role was 
to do additional assessments to determine students’ appropriateness for AoR. One mentioned 
that their scheduler selected all Level II students; another mentioned Level Is and IIs. 

On the other hand, some teachers, presumably those in regular English I courses, did find it 
appropriate to check EOG scores. A variety of criteria other than test scores were mentioned as 
playing a part in those decisions. Eight of the 63 respondents cited factors related to students’ 
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status as SWD or ESL. A few teachers commented that they made a decision based on students’ 
status alone. Others were more specific, indicating Individual Education Plans (IEPs), English 
proficiency based on ACCESS tests, or other scores drove their decisions within these groups. 

No data-based guidance has been available regarding which SWD or LEP students might benefit 
to date, so this evaluation may help inform those decisions. 

Eight teachers also mentioned the CORE assessments, which are a recommended step in the 
selection process. Four mentioned the AoR pre-assessment, which is a longer assessment 
designed for placement once students have been selected. 

Just over half of the respondents (54%) indicated they could serve all of the students who 
appeared to have needs based on their screening in 2009-10. Those who said they could not 
cited several capacity reasons: insufficient time (6), computers (7), space (5), teachers (5), and 
too many students (4). 

• Time often referred to scheduling issues, but also to the issues of the short length of time 
available to use at lunch or other times outside the normal school day. Two respondents 
mentioned that parents opted out of service to allow their students to take band or other 
classes. 

• Two teachers mentioned that the class was limited to use in a SWD class or an Integrated 
Reading class and therefore missed some non-SWD students who might benefit from AoR. 

Question: Did Teachers Implement the Program with Fidelity in 2009-10? 

Coaches’ Ratings 

The contracted coaches from TE21 (a company authorized to support AoR) observed teachers of 
the program several times a year, and rated them on whether they were meeting seven important 
objectives considered critical to program implementation on this scale: objective currently not 
being met (1), needs improvement (2), or being met (3). A copy of the form is attached as 
Appendix 2. A blank score was given if the coach was not able to observe the objective being 
implemented. 

For the 2009-10 cohort, coaches visited 209 teachers at times when AoR was scheduled to occur 
during the spring of 2010. They were actually able to observe in 168 classrooms (80%); AoR 
was not occurring in 41 classrooms on the day of their visit. Teacher absences and schedule 
changes were two reasons observations did not take place in 20% of the cases. For the 168 
teachers observed, the most common (modal) rating was a “3,” indicating appropriate program 
implementation. There were some cases where only a portion of the objectives were observable; 
this was particularly true for teachers who couldn’t be observed, but their students’ time on task 
and progress could be obtained by the coach. In these situations, averages were computed from 
the ratings of the observed objectives. Just over three-fourths were implementing six or all seven 
of the objectives. 

• Almost 60% of the teachers met all program objectives (97 of 168 or 57.7%), 


17 



Academy of Reading Evaluation 


E&R Report No. 10.23 


• 32 (19.0%) met 6 of 7 objectives, with a need for improvement in one area, 

• 39 (23.2%) needed improvements in more than one area, with 15 rated as a solid “2,” and 2 
teachers were rated as a “1.” 

Teachers’ Views 

A copy of the teachers’ survey, and the corresponding results from the multiple-choice and open- 
ended questions, are shown in Appendix 3. Overall, 63 responses were received. 

Training: All but two respondents indicated they had training in AoR. Over 90% of 
respondents either agreed or strongly agreed that the training helped them 
understand and implement the program reasonably well. Similarly, 90% indicated 
that the AoR coaching was essential to their ability to implement the program 
effectively. For all three items, “agree” was slightly more common than “strongly 
agree” as a response. 

Strategies: Results related to strategies suggest AoR was implemented appropriately. 

AoR was most commonly used in remedial electives in 2009-10 (44% of 
respondents), with Curriculum Assistance and regular English I classes the next most 
common (at about 15% each). Other class settings mentioned included reading 
decoding and Curriculum Assistance (for SWD students), ESL, and Occupational 
Course of Study (Item 3). 

Overall, 89% of teachers reported using AoR three to five days a week, which is 
recommended. The largest percentage of respondents used AoR three days a week 
(44%); 11% indicated they used AoR four days a week; 34% indicated daily use; 

11% utilized it two days a week (Item 10). 

Most schools were in the recommended range of session length of 30-40 minutes; 
39% indicated 30 minute sessions and 45% indicated 45 minute sessions. Small 
percentages indicated 25 minutes or less (11%) or 50 minutes or more (5%). WCPSS 
contacts indicate 40 minutes (or more) is preferable because it allows more time for 
instruction since set up and closing out the sessions take the same amount of time 
regardless of session length (Item 11). 

Likes: In general, many teachers liked aspects of the use of technology, program content, 

pacing, motivators used, and data for teachers and students to monitor progress. 

Since many comments were repeated, only representative open-ended comments are 
included in Appendix 3b. 

Technology — Many respondents liked the fact that AoR uses the computer and that 
it is interactive. A few mentioned liking the graphics. Many respondents mentioned 
the value of the reporting tools available for both students and teachers to monitor 
progress. Several teachers specifically mentioned the value of the teacher tool in 
allowing them to monitor progress and support students one-on-one while others 
continue learning. 
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Likes (continued) 

Content — Many teachers liked the program’s focus on basic skills. Word building, 
phonics, phonological awareness, visual-sound matching, automaticity and fluency, 
and oral distinctions of letters were mentioned specifically. One teacher said she 
“saw gains from students who were compensating in reading by skipping 
words... [students] actually slow down and try to work out the words.” 

Pacing — Numerous teachers appreciated the fact that AoR allowed students to work 
at their own pace and level. Teachers also noted that the program allows students to 
focus just on their weak areas. A few teachers mentioned liking the fact that 
students must master needed skills before progressing. 

Motivation — The certificates and other motivators used were also mentioned as 
positive. One teacher commented that “students work hard to earn them. Students 
remained focused and are motivated to reach the next level.” Another wrote that the 
format made students “less embarrassed about doing foundational work.” Others 
indicated it was good with motivated students, with middle school students, and with 
ESL students. A few teachers mentioned that using the computer was motivating in 
itself. 

Dislikes: Five or more teachers mentioned dislikes that fell into most of the same categories. 

Technology — At least five respondents mentioned computer glitches such as 
incorrect scoring of responses, unreliable networks, and charts that would not print. 
Another issue mentioned numerous times was that the narrator’s voice was difficult 
to understand (i.e., a “Canadian accent” with different pronunciation of some letter 
sounds). The volume was cited as inconsistent by some respondents. One teacher 
expressed disappointment with the reporting tools. 

Content — The lack of enough comprehension material was cited by nine teachers. 
Vocabulary was also cited as an area to add. Others felt the materials were too 
childish for high school students, including some of the logos, illustrations, and 
visuals. 

Pacing — Allowing students to self-pace themselves was cited as a plus by some 
teachers, but the pacing of the program was described as boring, repetitious, 
frustrating, and tedious by over 10 respondents. One respondent’s comment on 
timed lessons was illustrative: “Kids get stuck on the pacing. So even though they 
may have achieved accuracy for 50 trials, they need to keep doing it until the pacing 
is the same.” 
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Dislikes (continued): 

Motivation — Motivating students to buy into the program 
was described as difficult because the program was viewed 
as boring. Comments suggested some students were just 
bored with school in general; others said it was related to 
students getting stuck on a skill which held them back 
from moving forward. 

Class Size — Most teachers did not cite class size as an 
issue. However, two teachers indicated their classes were 
too large to allow as much one-on-one work as they would 
like. Central staff recommends 6-10 students per teacher 
(depending on the number of special education students). 

Challenges: Appendix 3c includes representative open-ended 

comments about challenges encountered and ways the 
program could be improved. 

Technology — Computer access and dependability of the 
network were described as technology hurdles for some 
schools. Two described computer crashes during student 
training time. One teacher indicated that, “for a while, the 
program would just not work correctly on Thursdays.” 

Not having enough computers to handle all the children 
who might benefit was also mentioned by two schools. 


Challenges 

Technology: “At our 
school, no one was 
interested in helping 
with the computer 
problems. It was hit or 
miss whether we could 
use them or not. It 
made for planning 
nightmares, students off 
task because of a 
change in the routine, 
and the data wasn ’t as 
valid as it should have 
been. ” 

Content: “AoR is not 
the best program to be 
used with students with 
severe reading 
problems. I 
supplemented AoR with 
phonics games, tests, as 
well as reading 
comprehension stories 
that were on the level 
of my students. ” 



Content — Concerns related primarily to boredom with the 
tedious parts of the program and the same pacing issue 
(being able to answer items correctly within a certain time 
limit). One teacher said, “having 100 plus trials is not 
uncommon for some students.” Another teacher indicated 
that the listening-visual match part is very confusing and 
the slightest noise throws students off. Finally, one teacher 
indicated she had students in her group who did not know 
enough English to understand the program or her directions. 


Pacing and Motiva- 
tion: “The program is 
not for everyone. Some 
kids grow so frustrated 
with the program and 
start hitting keys 
instead of trying. ” 


J 


Pacing and Motivation — These issues again went hand in hand, with teachers 
describing difficulties related to getting students to buy-in and to remain motivated 
to continue the program. One teacher indicated her students “hate the program and 
are frustrated by it.” Finally, one teacher indicated, “High school students 
sometimes rebel at having a reading class.” 


Other issues came up related to teacher and scheduling issues. One teacher indicated 
she/he had too many students and not enough support. She/he had to develop 
additional work for those who completed the program and it was overwhelming for 
him/her. 
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Ideas for Improvement: In terms of ideas for improvement, most 
suggestions related to the same areas. 


Improvements 


Technology — Eight teachers made suggestions for 
technology improvements that focused on: 

• eliminating glitches in the software, 

• expanding program capabilities, 

• making the interface more user-friendly so 
that more teachers could participate, 

• creating a smarter interface for recognizing 
mastery of a skill, 

• having sounds that were pronounced more 
accurately, consistently, clearly, and loudly 
every time, 

• adding enhancements so that teachers could 
adjust learning streams or specific 
assignments, and see more from their 
station, 

• adding an option for students to be able to 
have a sound repeated and to be able to see 
their own reports so they could monitor their 
own progress. 

Content — Many respondents requested more focus on 
comprehension and vocabulary development. Several 
teachers requested more mature characters be used for 
middle and high school students — especially high 
school students. 


Technology: I would 
like what the teacher 
can "see" as the student 
responds to trials to be 
expanded. On the 
comprehension, I would 
like to "see " the stories 
without having to read 
over the student's 
shoulder, or to see the 
responses to auditory/ 
visual exercises without 
having to use a splitter 
to interrupt the student 
as he/she is 
progressing. 

Content: Teachers need 
to be able to make 
adjustments to the 
student's lesson 
assignments as needed. 

For example, I teach 
9th grade students, 
many of whom need 
lessons from the 6/7 
Comprehensive 
Training Stream, 
however the computer 
assigns them to the 8th 
grade stream. 

J 


Pacing/Motivation — A number of teachers suggested that the time/pacing 
requirements be relaxed because students get stuck on a skill otherwise. As one 
teacher commented, “Take away the demand that students complete skill sets within 
a specific time limit or within a consistent time limit.” Findings ways to help 
students reach the core readings more quickly was specifically mentioned. 


Structures — Two teachers noted that some students may need more than a year to 
complete the program and this option should be available. One teacher felt the class 
worked better as an elective. One suggested that it be changed to a direct instruction 
program four days a week during the summer. 
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Question: Did the Program Meet Its Short-Term Goal for Program Completion? 

AoR includes a pre-assessment of students’ skill level, with students classified as Below Basic 
(roughly two years or more below grade level), Basic (about one year below grade level), 
Proficient (at grade level), Advanced (above grade level), or Graduate (well above grade level). 
Graduates have mastery of all skills taught through AoR. Based on the pre-assessment, the 
lowest scoring students are assigned the greatest number of skills to complete, with the opposite 
being true for the higher scoring students. The vendor has indicated that most students should 
finish in one semester, but that students starting at the Below Basic level may need two 
semesters or more to complete the full training stream. SWD and LEP students may also need 
extra time to reach automaticity. In WCPSS, students can typically have one or two semesters in 
the program. 

The WCPSS goal for the program is for all students to master 80% or more of the skills in their 
assigned training stream. This is a high standard for success; gaining two categories (or grade 
levels) is another standard for success seen in research studies (Loh, 2009). 

Completion Rates Overall and by Pretest Levels 

Table 6 illustrates that only 25.6% of students with pre-tests completed 80% or more of their 
training stream in 2008-09 and 40.8% in 2009-10 (see the Total row). While the improvement 
from 2008-09 to 2009-10 is notable, AoR fell far short of the district goal for AoR service both 
years, which was that all students would complete at least 80% of their training stream. 

The Total Enrolled column in Table 6 illuminates a key contributing factor, which is that the 
most common pre-assessment level by far was Below Basic in both 2008-09 and 2009-10 (about 
three-fourths of the students). Thus, most of those served had a large number of skills to 
complete. A distant second was Basic, which included 13% and 18% of the 2008-09 and 2009- 
10 students, respectively. Proficient was the next most common category of students. TE21 
trainers consider these three categories of students as likely to benefit from the AoR program. 
About 5% of the students served who took a pre-test (67 in 2008-09 and 149 in 2009-10) scored 
advanced or graduate, which TE21 trainers indicate is too high to benefit greatly from the 
program. These students would only be assigned a few of the more advanced skills to work on 
to complete their training stream. Completion rates varied by year of service and pre-test level. 

• As the italics indicate, students served in 2009-10 were showing higher completion rates for 
each pretest level than for those served in 2008-09. The groups with the highest completion 
rate of assigned work were those within the higher categories initially with the least work to 
complete — 63% to 72% of those scoring proficient, advanced, or graduate initially in 2009- 
10. Still, about one in three students in these groups did not complete their assigned training 
stream. 

• For those students who initially scored Below Basic, only 20% completed their training 
stream in 2008-09, and 36% in 2009-10. Thus, four in five of these students in AoR in 
2008-09, and two in three in 2009-10, failed to complete their training stream. 
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Table 6 

Pre Test Levels for Students in AoR 
and Percent Completing at Least 80% of their Training Stream 



2008-09 

2009-10 


Total 

Enrolled 

Completing 

Total 

Enrolled 

Completing 

Pre Test Level 

Number 

Percent 

Number 

Percent 

Below Basic 

1,078 

218 

20.2% 

2,720 

982 

36.1% 

Basic 

179 

82 

45.8% 

639 

314 

49.1% 

Proficient 

34 

12 

35.3% 

131 

82 

62 . 6 % 

Advanced 

18 

7 

38.9% 

53 

36 

67 . 9 % 

Graduate 

49 

29 

59.2% 

96 

69 

71 . 9 % 

Total 

1,358 

348 

25.6% 

3,639 

1,483 

40.8% 


Note: One student did not take the pre-test in 2009-10. 

Interpretation Example: 1,078 of the 1,358 students (79%) of those served in AoR in 2008-09 scored Below 

Basic on their pre-assessment; 20% of the 1,078 students (218) completed 80% or more of their 
training stream. 


Final AoR Categories by Completion Rate 

Those who completed their training stream were more likely to be classified as Graduate based 

on a final post-assessment (See Figure 4.) 

• A total of 552 students (15.2%) did not take a post-assessment and therefore could not be 
assigned a final status. Most did not complete 80% of their training stream, and we assume 
nearly all would not have reached graduate status. 

• Of the 1,483 who completed at least 80% of their training stream, 20.4% (302) reached the 
status of Graduate. Below Basic and Basic were the most common final statuses (See Figure 
4.) 

• Of the 2,157 students who did not complete 80% of their training stream, 102 (4.7%) reached 
the status of Graduate. 

• Overall, 1 1% (404) of AoR 2009-10 participants reached Graduate status. 
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Figure 4 

Final Status of 2009-10 AoR Participants by Completion Status and Overall 



Note: Students who completed 80% or more n=l,483. Students who did not complete at least 80% n=2,157. 
Total N (last column) represented all AoR students in 2009-10 (3,640) 


If we focus only on students completing at least 80% of their program and taking a posttest, 
Table 7 indicates that only about one quarter of the AoR students reached the goal of Graduate 
(26.5% in 2008-09 and 21.1% in 2009-10). Students’ chances of reaching Graduate status 
generally increased as the students’ initial category increased. 

• Those who started out as Below Basic (the largest group) were most likely to stay in that 
category in both 2008-09 and 2009-10. For example, in 2009-10, only 15.5% of those 
starting out as Below Basic were able to reach the Graduate category. If improvement of two 
or more categories is checked, 29.8% of the students met this standard for success. 

• Those who started out as Basic were most likely to stay as Basic (35.9% and 38.9%). In 
2008-09, almost as many (33.3%) moved up to Graduate status. An additional 7.7% reached 
the Advanced category, meaning 41% improved by two categories. In 2009-10, 38.5% 
increased by two categories. 
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• Those who started out as Proficient had a varied pattern in 2008-09 and 2009-10. (Note only 
1 1 students were in this category in 2008-09—11.) For 2008-09 participants, students were as 
likely to increase two categories to Graduate or to decrease two categories to Below Basic 
(27.3% each). In 2009-10, more students moved up to Graduate (32.9%), than down to 
Below Basic (12.7%). 

• Those who started out as Advanced or Graduate scored above grade level initially and had 
few skills assigned. The highest percentage of students stayed in their initial category, with 
about one third of the Advanced students moving up to Graduate in 2009-10. However, 
some students dropped in category status, especially in 2009-10. 

Thus, students scoring Below Basic initially were unlikely to improve to the Graduate level, even 

if they completed 80% of their training stream. Moving up two categories was also unlikely. 

The likelihood of improving to Graduate was greater in other initial categories. 


Table 7 

Pre- and Post- Test Levels for Students Completing 80% or More of the Program 


2C 

108-09 Pre- and Post-Test Levels 

Post 

Below Basic 

Basic 

Proficient 

Advanced 

Graduate 

Total 

Pre 

n 

% 

n 

% 

n 

% 

n 

% 

n 

% 


Below Basic 

118 

57.8% 

43 

21.1% 

9 

4.4% 

5 

2.5% 

29 

14.2% 

204 

Basic 

11 

14.1% 

28 

35.9% 

7 

9.0% 

6 

7.7% 

26 

33.3% 

78 

Proficient 

3 

27.3% 

2 

18.2% 

2 

18.2% 

1 

9.1% 

3 

27.3% 

11 

Advanced 

* 

•k 

* 

* 

k 

* 

k 

* 

k 

* 

7 

Graduate 

1 

3.6% 

2 

7.1% 

0 

0.0% 

0 

0.0% 

25 

89.3% 

28 

Total 

134 

40.9% 

75 

22.9% 

18 

5.5% 

14 

4.3% 

87 

26.5% 

328 

2C 

109-10 Pre- and Post-Test Levels 

Post 

Below Basic 

Basic 

Proficient 

Advanced 

Graduate 

Total 

Pre 

n 

% 

n 

% 

n 

% 

n 

% 

n 

% 


Below Basic 

464 

48.5% 

207 

21 .7% 

85 

8.9% 

52 

5.4% 

148 

15.5% 

956 

Basic 

18 

6.1% 

115 

38.9% 

49 

16.6% 

42 

14.2% 

72 

24.3% 

296 

Proficient 

10 

12.7% 

18 

22.8% 

15 

19.0% 

10 

12.7% 

26 

32.9% 

79 

Advanced 

1 

2.9% 

3 

8.6% 

5 

14.3% 

15 

42.9% 

11 

31 .4% 

35 

Graduate 

6 

9.1% 

10 

15.2% 

5 

7.6% 

0 

0.0% 

45 

68.2% 

66 

Total 

499 

34.8% 

353 

24.7% 

159 

11.1% 

119 

8.3% 

302 

21.1% 

1,432 


Note 1: In 2009-10, 51 additional students completed at least 80% of their training stream, but did not take pre- or post-assessment. 
Note 2: In 2008-09, 20 additional students completed at least 80% of their training streams, but did not take a post-assessment. 
Note 3: No detail is provided for group sizes less than 10 to support confidentiality of student data. 

Note 4: Italics show most common post-placement for each pre-assessment group. 

Interpretation Example: For students scoring Below Basic upon entry to AoR in 2009-10, almost half (48.5%) scored at the same 
level on the post-assessment; 15.5% showed great improvement by reaching graduate status. 
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Goal Attainment EOG/EOC Impact 

Analyses of outcomes of AoR on EOG/EOC achievement focus on 
the 2009-10 cohort over time (before and after service). The 
program was more mature and more students were available for 
analysis in the 2009-10 cohort. We will always use the word 
“cohort” and the test dates rather than school years to distinguish 
these analyses over time from those reflecting the separate groups 
served in 2008-09 and 2009-10. 

As noted earlier, less than half (41%) of the 2009-10 cohort in AoR completed 80% or more of 
their training stream. When we examined AoR program completion based on prior EOG/EOC 
scores, we found some groups were more successful in completing the program than others. 

(See Table 8.) 

• Students in the 2009-10 cohort were most likely to complete their training stream (about 
53%) if they initially scored in the upper range of Level II or the bottom range of Level III on 
the EOG/EOC in spring 2009. This is the range WCPSS central staff had recommended for 
service. 

• Several groups had completion rates between 42% and 48% — those who scored in the upper 
part of Level I (42%), the bottom or middle part of Level II (44% and 48%), or the middle 
part of Level III (48%). These hovered around the overall completion rate of 45%. 

• Those with more extreme scores — those scoring in the bottom part of Level I or the upper 
part of Level III or IV initially — were less likely to complete the program (25-33%). 

Table 8 

2009-10 Cohort Program Completion by Prior EOG/EOC Achievement Level Range 



Students Completing 80% or 
More of the Program 

Total 

Students 

Achievement Level 

n 

% 

n 

Level I - Bottom 

* 

* 

1 

Level I - Middle 

116 

30.1% 

385 

Level I - Upper 

285 

41 .7% 

683 

Level II - Bottom 

133 

43.6% 

305 

Level II - Middle 

342 

48.0% 

712 

Level II - Upper 

254 

52.8% 

481 

Level III - Bottom 

149 

53.4% 

279 

Level III - Middle 

70 

47.9% 

146 

Level III - Upper 

3 

25.0% 

12 

Level IV - All 

* 

* 

9 

Total 

1,355 

45.0% 

3,013 


Note 1 : There were 246 students who took alternate assessments the prior year, 37 
(15%) of whom completed the program. Similarly, there were an additional 
381 students with missing EOG/EOC scores, 91 (23.9%) who also 
completed the program. 

Note 2: Shading and italics reflects the most positive results. 


r n 

The word “cohort” always 
refers to the AoR students 
sensed in 2009-10 over 
time (from before to after 
AoR service). 
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Another question of interest is the relationship between prior EOG/EOC levels and the AoR 
category assigned to students before and after service. This was studied for those completing at 
least 80% of their training stream within the 2009-10 cohort (see Table 9). Higher prior 
EOG/EOC scores did relate somewhat to higher pre- and post-assessment AoR scores, but 
assessment category placements were lower than expected. 

• Overall, most students who were assessed for AoR started out as Below Basic, which TE21 
coaches described as two years below grade level. This was also the most common 
placement for students from the middle of Level I to the middle of Level III on the prior 
year’s EOG (the only groups with adequate sample sizes for analysis). The percentage of 
students in Below Basic declined with higher prior EOG/EOC scores, with a range of 82% to 
45%. These findings suggest that the pre-assessment is very difficult. Some relationship of 
EOG/EOC scores is evident, but some higher- achieving students also may not have done 
their best on the pretest. 

• At the end of the 2009-10 school year after service, the most common result was that those in 
the bottom three EOG/EOC Level score categories continued to stay in the Below Basic 
category. Those in the middle of Level II were most likely to move up one category to Basic, 
and those above this point on EOG/EOC scores were most likely to reach Graduate status. 

• The decline in the percentage of students in the Below category varied by EOG/EOC pre-test 
level, with those starting in Level II or low to middle Level III having the greatest reduction 
of 35% (compare the first column of the table from the Pre to Post EOG Level). Those with 
higher EOG/EOC pre-levels had a slightly higher reduction than those with EOG pre-levels 
in Level I. 


27 



Academy of Reading Evaluation 


E&R Report No. 10.23 


Table 9 

2009-10 Cohort AoR Assessment Levels Before and After Service 
based on Prior EOG/EOC Scores 



Pre AoR Level 

Pre EOG/EOC 
Level 

Below 

Basic 

Proficient 

Advanced 

Graduate 

Total 

n 

% 

n 

% 

n 

% 

n 

% 

n 

% 

n 

Level I - Bottom 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Level I - Middle 

91 

82.0% 

18 

16.2% 


0.0% 



2 

1 .8% 

111 

Level I - Upper 

206 

74.1% 

55 

19.8% 

8 

2.9% 

6 

2.2% 

3 

1.1% 

278 

Level II - Bottom 

98 

75.4% 

21 

16.2% 

7 

5.4% 

2 

1 .5% 

2 

1 .5% 

130 

Level II - Middle 

219 

65.8% 

71 

21 .3% 

23 

6.9% 

5 

1 .5% 

15 

4.5% 

333 

Level II - Upper 

139 

57.0% 

58 

23.8% 

20 

8.2% 

11 

4.5% 

16 

6.6% 

244 

Level III - Bottom 

80 

54.8% 

39 

26.7% 

8 

5.5% 

6 

4.1% 

13 

8.9% 

146 

Level III - Middle 

30 

44.8%, 

17 

25.4% 

10 

14.9% 

3 

4.5% 

7 

1 0.4% 

67 

Level III - Upper 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

3 

Level IV - All 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

2 

Total 

864 

65.8% 

281 

21 .4% 

76 

5.8% 

33 

2.5% 

60 

4.6% 

1,314 

Pre EOG/EOC 
Level 

Post AoR Category 

Level I - Bottom 

* 

* 

* 

* 

* 

k 

k 

* 

k 

* 

* 

Level I - Middle 

67 

60.4% 

22 

1 9.8% 

6 

5.4% 

6 

5.4% 

10 

9.0% 

111 

Level I - Upper 

143 

51.4% 

61 

21.9% 

27 

9.7% 

23 

8.3% 

24 

8.6% 

278 

Level II - Bottom 

52 

40.0% 

34 

26.2% 

20 

15.4% 

9 

6.9% 

15 

1 1 .5% 

130 

Level II - Middle 

87 

26.1% 

108 

32.4% 

37 

11.1% 

25 

7.5% 

76 

22.8% 

333 

Level II - Upper 

47 

1 9.3% 

60 

24.6% 

40 

16.4% 

32 

13.1% 

65 

26.6% 

244 

Level III - Bottom 

28 

1 9.2% 

35 

24.0% 

16 

1 1 .0% 

16 

1 1 .0% 

51 

34.9% 

146 

Level III - Middle 

11 

16.4% 

13 

1 9.4% 

7 

10.4% 

5 

7.5% 

31 

46.3% 

67 

Level III - Upper 

★ 

•k 

★ 

k 

k 

k 

k 

k 

k 

k 

3 

Level IV - All 

★ 

k 

★ 

k 

k 

k 

k 

k 

k 

k 

2 

Total 

435 

33.1% 

334 

25.4% 

154 

1 1 .7% 

116 

8.8% 

275 

20.9% 

1,314 


Note: Analysis is based on those completing 80% or more of the program and having pre- and post-assessments in 
AoR. An additional 5 1 students completed at least 80% of the program but did not take the post-assessment. 
Also, 37 students took alternate assessments and 91 had no scores at all. The mode ( the most common value) is 
highlighted in italics for each row. 

Interpretation Example: Of the 244 2009-10 AoR students who scored in the upper Level II range on their 2008-09 
EOG/EOC, 57% pre-tested in the “Below” (Basic) category in AoR. On the post-assessment, 19.3% of these 
students remained in the “Below” category as well, with 37.7% moving up to higher categories. 
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Question: Did AoR Meet Its Short-Term Goal for EOG/EOC Impact? 

A second short-term goal for WCPSS staff was for AoR participants to increase their growth on 
the North Carolina End of Grade and End of Course tests. This was measured through Academic 
Change (AC) scores on the tests, which measure whether students made at least one year’s 
growth for a year of instruction relative to the rest of the state’s population. This measure is 
more sensitive to growth within level score ranges, and is a better measure of overall growth 
across the group. E&R staff analyzed both the percentage of students reaching AC growth 
targets and the changes in average AC scores from year to year and compared them to the state 
data. 

Percentage of Students Reaching or Exceeding Academic Change Growth Targets 

Table 10 shows several important positive trends for EOG/EOC 
result patterns for the 2009-10 cohort. 

• For the 2009-10 cohort of students in AoR, the percentage of 
participants reaching their growth targets increased 
considerably, from 37.5% to 60.1%. 

• The percentage of students in each grade who reached their 
reading/English growth target on the EOG/EOC increased at 
each grade except Grade 10 (where the number of students was 
small). 

• Utilization of AoR at the middle school level appears to be somewhat more likely to lead to 
student success in reaching EOG/EOC growth targets than at the high school level, with the 
highest increases and overall percentages of students reaching growth targets. Grade 7 had 
the largest improvements (increasing 33 percentage points, from 30% to 63% reaching their 
AC target). 


r ■> 

The percentage of 
participants reaching 
their growth targets 
improved considerably 
from before to after AoR 
service, especially at the 
middle school level. 
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Table 10 

Percentage of Students in the 2009-10 Cohort of AoR Who Reached 
or Exceeded Growth Targets in 2008-09 and 2009-10 on EOG/EOCs 



Spring 2009 

Spring 201( 


Grade in 
2009-2010 

Met Target 

Total 

Met Target 

Total 

n 

% 

n 

n 

% 

n 

6 

330 

39.7% 

831 

551 

60 . 7 % 

907 

7 

195 

30.0% 

651 

468 

62 . 8 % 

745 

8 

293 

40.9% 

717 

486 

60 . 3 % 

806 

9 

151 

37.4% 

404 

242 

54 . 5 % 

444 

10 

18 

62.1% 

29 

2 

33.3% 

6 

11 

* 

* 

<10 

- 

- 

0 

12 

. 

- 

- 

- 

. 

0 

All 

988 

37.5% 

2,634 

1,749 

60 . 1 %, 

2,908 


Note: Of the 3,640 participants in 2009-10, 2,634 had academic change 

scores in 2008-09 and 2,908 in 2009-10. Reaching a growth target for 
a student is defined as having an academic change score greater than or equal to zero. 

An increase in the percentage of students with positive AC scores is highlighted and italicized. 

The percentage of students meeting their AC growth target on the EOG/EOC increased for 
students regardless of the teachers’ level of implementation of AoR. The increase was slightly 
greater for those in the classrooms of teachers with middle or high levels of implementation than 
those with low levels of implementation. It is interesting that those “not observed” had the 
highest percentage of students reaching growth targets in 2009-10, but the reason for this result is 
not known. 


Table 11 

Implementation Level and 2009-10 Cohort Students Meeting or 
Exceeding Growth Targets 



Spring 2009 

Spring 2010 

Implementation 

Level 

Met Target 

Total 

Met Target 

Total 

n 

% 

n 

n 

% 

n 

Low 

178 

39.6% 

449 

277 

55.2% 

502 

Middle 

151 

36.7% 

412 

267 

59.7% 

447 

High 

494 

36.7% 

1,346 

884 

59.4% 

1,489 

Not Observed 

165 

38.6% 

427 

321 

68.3% 

470 

All 

988 

37.5% 

2,634 

1,749 

60.1% 

2,908 


Note: Implementation ratings came from AoR coach evaluation forms. Not all teachers were available 
to be rated at the time the coach visited the school. Also, based on classroom names, not all 
teachers received documented coaching visits, making their implementation level unable to be 
determined. 
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Tables 12 and 13 show the patterns of growth for the 2009-10 cohort based on students’ pre- 
assessment levels and post-assessment levels. When patterns of 
growth are examined based on students’ initial and final 
placement in the program, positive trends are evident. 


The number of students in the lower categories far 
outweighs those in the higher categories. The greatest 
increase in number of proficient students was within the 
Below Basic group. 

All categories of students showed an increase in the 
percentage of students reaching growth targets between 
Spring 2009 and Spring 2010. 


The greatest increase in 
number of proficient 
students was within the 
Below Basic group. 

Those initially scoring in the 
highest categories ofAoR 
showed the greatest 
improvement in the 
percentage of students 
making growth over time. 


Those placed in the highest categories of AoR initially 
showed the greatest improvement in the percentage of students making growth over time 
(with the percentage of those classified as Graduate increasing from 28% reaching EOG AC 
targets to 72%). 


When patterns for growth targets are examined relative to post-test levels after service in AoR, 
patterns are similar but somewhat less in magnitude. 


Table 12 

Percentage of 2009-10 Cohort Meeting or Exceeding Growth Targets 

by Pre-Level AoR 



Spring 2009 

Spring 2010 

Pre Level 

Met Target 

Total 

Met Target 

Total 

n 

% 

n 

n 

% 

n 

Below Basic 

730 

39.0% 

1,872 

1,193 

57.4% 

2,080 

Basic 

185 

35.0% 

529 

374 

65.7% 

569 

Proficient 

38 

33.3% 

114 

85 

68.5% 

124 

Advanced 

14 

32.6% 

43 

36 

72.0% 

50 

Graduate 

21 

27.6% 

76 

61 

71 .8% 

85 

All 

988 

37.5% 

2,634 

1,749 

60.1% 

2,908 


Note: 988 of the 3,639 students that took a pretest in 2009-10. 

Additionally, 2,908 of these students had academic change scores in 2009-10. 
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Table 13 

Percentage of 2009-10 Cohort Meeting or Exceeding Growth Targets by Post Level AoR 



Spring 2009 

Spring 2010 

Post Level 

Met Target 

Total 

Met Target 

Total 

n 

% 

n 

n 

% 

n 

Below Basic 

359 

36.8% 

976 

585 

52.8% 

1,107 

Basic 

211 

37.9% 

556 

385 

64.0% 

602 

Proficient 

83 

34.4% 

241 

166 

63.4% 

262 

Advanced 

52 

31.7% 

164 

137 

73 . 3 % 

187 

Graduate 

136 

41 .2% 

330 

261 

72.7% 

359 

All 

988 

43.6% 

2,267 

1,749 

69.5% 

2,517 


Note: 988 students that post tested had academic change scores in 2008-09; 2,517 in 2009-10. 


The question of whether completing 80% of the program or more was critical to success on 
EOG/EOC growth is addressed in Table 14. 

• Student groups who completed 80% or more and those who completed less than 80% both 
showed improvement in terms of the percentage of students reaching growth targets. 

• The greatest improvement was evident for those who completed 100%. The differences by 
completion subgroup were only about 11%. 

Table 14 

Percentage of 2009-10 Cohort Meeting or Exceeding Growth Targets by Completion Rate 



Spring 2009 

Spring 2010 

Program Completion 
in 2009-10 

Met Target 


Met Target 


n 

% 

Total 

n 

% 

Total 

Completed 100% 

312 

36.7% 

851 

622 

66 . 8 % 

931 

Completed 80-99% 

132 

36.7% 

360 

245 

62.7% 

391 

Completed 80% or More 

444 

36 . 7 % 

1,211 

867 

65 . 6 % 

1,322 

Completed Less Than 80% 

544 

38.2% 

1,423 

882 

55.6% 

1586 

All 

988 

37.5% 

2,634 

1,749 

60.1% 

2,908 


Note: 988 students had academic change scores in 2008-09; 2,908 in 2009-10. 


Table 15 reveals several positive trends based on prior EOG level scores and growth for the 
2009-10 AoR cohort. 

• All EOG/EOC subgroups showed an increase in the percentage of students reaching their 
growth targets except those in the small group scoring in the upper level III category. 
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• The most positive change in the percentage of students reaching growth targets from 2008-09 
to 2009-10 was for students initially scoring in the middle of Level I. This group moved 
from 12.7% reaching their growth target to 55.7% reaching the growth target, an increase of 
43%. The group initially scoring in the upper range of Level I had the next best 
improvement (of approximately 33 percentage points). 

• The percentages of students reaching their growth targets increased between lower and 
higher prior EOG/EOC level scores in 2008-09. Differences in the percentage of students 
reaching growth declined in Spring 2010 after the AoR intervention. 

• Overall, the AoR Students moved up from below to at average district growth on the 
EOG/EOC, helping to close this achievement gap. In WCPSS overall, 62% of English I 
students and 59% of middle school reading students met their growth targets. 

Table 15 

Percentage of 2009-10 Cohort Students Meeting or Exceeding Growth Targets 

by 2008-09 EOG Level 



Spring 2009 

Spring 2010 

08-09 EOG/EOC 
Ach Level 

Met Target 

Total 

Met Target 

Total 

n 

% 


n 

% 


Level I - Bottom 

* 

* 

* 

* 

* 

* 

Level I - Middle 

37 

12.7% 

292 

192 

55.7% 

345 

Level I - Upper 

151 

25.5% 

592 

383 

58 . 4 % 

656 

Level II - Bottom 

100 

37.2% 

269 

174 

58.4% 

298 

Level II - Middle 

284 

45.5% 

624 

393 

56.3% 

698 

Level II - Upper 

210 

47.3% 

444 

294 

62.4% 

471 

Level III - Bottom 

123 

47.3% 

260 

201 

73.9% 

272 

Level III - Middle 

72 

53 . 3 % 

135 

95 

66.9% 

142 

Level III - Upper 

6 

54 . 5 % 

11 

5 

50.0% 

10 

Level IV - All 

* 

* 

7 

* 

* 

8 

All 

988 

37.5% 

2634 

1745 

60.2% 

2901 


Note: Italics indicate the highest percentage improvement in students reaching their growth 

targets between Spring 2008 and Spring 2009. 

Interpretation Example: In Spring 2009, the range of students in each pretest group reaching their 
growth standard was 12.7% (Level I Middle) to 54.5% (Level III Upper). The percentage 
of students in each pretest group meeting growth targets increased in Spring 2010, with 
differences between groups decreasing; the range was reduced to 50% (Level III Upper) 
to 73.9% (Level III Bottom). 

Analyses of the percentage of students making growth by demographic category for 2009-10 can 
help identify student groups most likely to benefit from AoR. Of course, staff must be aware 
that individuals within groups may or may not fit the group profile and that additional testing and 
information should inform their decisions. Also, trends could vary across cohorts, especially for 
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groups with small numbers of students served. SWD and LEP students improved overall in the 
percentage reaching growth targets, but not to the same extent as all students. Results suggest 
some types of SWD and English as a Second Language (ESL) students are more likely to benefit 
from AoR than others. 

• SWD students who are in the least restrictive settings (regular classrooms most of the time) 
are more likely to benefit from AoR than those in resource or separate settings. 

• SWD students who were autistic or had serious emotional difficulties appeared to be more 
likely to improve than learning disabled students. 

• ACCESS scores (not shown) suggest LEP students with some English but not extensive 
English are most likely to benefit from AoR (scores of 3 or 4 on a 6 point scale). The 
percentage of ESL students who reached their growth targets increased by only 8% after AoR 
service, with a higher increase for LEP students (of 15 percentage points). 


Table 16 

Percentage of 2009-10 AoR Students Making Growth Targets 
Before and After Service by Demographic Group 




2008-09 

2009-10 



% 

n 

% 

n 

Not 

SWD 

Not SWD/AG 

37.2% 

1930 

62.5% 

1365 

Academically Gifted 

22.6% 

53 

74.5% 

41 

All 

36.8% 

1,983 

62.8% 

1,406 

SWD 

SWD Setting 





Regular 

38.0% 

413 

54.7% 

437 

Resource 

43.0% 

207 

46.0% 

202 

Separate 

27.8% 

18 

35.7% 

14 

Separate School 

* 

<10 

50.0% 

10 

SWD Group 

2008-09 

2009-10 

Autistic 

50.0% 

12 

71 .4% 

14 

Learning Disabled 

40.8% 

387 

48.9% 

401 

Other Health Impaired 

39.4% 

198 

53.8% 

195 

Serious Emotional Disability 

27.7% 

36 

55.0% 

40 

All SWD 

39.6% 

651 

51 .3% 

688 


ESL/LEP Status 

2008-09 

2009-10 

ESL 

ESL 

44.7% 

179 

52.8% 

216 

Not ESL 

37.0% 

2,452 

60.7% 

2,686 

LEP 

LEP 

43.6% 

388 

59.2% 

439 

Not LEP 

36.5% 

2,246 

60.3% 

2,469 


Note: Groups and settings with fewer than 10 students are not included in this table, including the hearing impaired, 
intellectually disabled, students with speech/language issues, and those in a home setting. 
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Average Academic Change 

The state’s ABCs academic change (AC) score reflects whether students as a group grew more 
or less than the target projection. A growth score of zero means the target was met exactly. 
Tables 17 through 20 depict the mean and range (minimum and maximum values) of the 
academic change score for students enrolled. The AC for Spring 2009 actually reflects the 
growth between Spring 2008 to Spring 2009. The AC for Spring 2010 reflects student growth 
between Spring 2009 and 2010. Positive changes are noted in the Difference column. 

• Across the tables, nearly all pretest means for Spring 2009 and Spring 2010 were negative 
while all post-test score means increased and were positive for EOG groups (with the 
exception of upper Level III students). 

• By prior EOG/EOC achievement level, the difference columns show that the mean academic 
change scores were all positive, ranging from 0.1 (for upper Level III students) to 0.67 for 
middle Level I students. The mean difference was 0.29 across groups. Students in the 
middle and upper parts of Level I had above average improvements in AC mean scores. 

• The percentages of students reaching growth targets varied considerably both before and 
after service in AoR (see Min and Max columns). Variation actually increased after service. 

Tables 18, 19, and 20 show similar trends for EOG change scores based on the percent of the 
program completed, as well as the students’ initial and post-level performance in the AoR 
program. Improved AC scores were evident from one year to the next for all subgroups, 
changing from below the state expectation to above it. The difference between AC mean scores 
before and after service was 0.29 — more than half a standard deviation (which was 0.5 on this 
measure in 2009-10). The groups that exceeded this average difference were those who 
completed 80% or more of the program, and those in the Basic level or above before or after 
service. 

Considerable variation in average academic change was evident at a teacher level, ranging from 
-0.77 to +0.75. At a school level, it ranged from -0.46 to +.95. Thus, in both cases, some had 
well below average achievement while others had above average representing over one SD of 
improvement. 
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Table 17 

Academic Change Score Statistics for 2009-10 Cohort of Participants by 2008-09 EOG/EOC Achievement Levels 


2008-09 Achievement 
Level 

Spring 20 

09 

Spring 2C 

10 

Difference 

Min 

Max 

Mean 

SD 

n 

Min 

Max 

Mean 

SD 

N 

Min 

Max 

Mean 

SD 

n 

Level I - Bottom 

- 

- 

- 

- 

0 

* 

* 

* 

- 

1 

- 

- 

- 

- 

0 

Level I - Middle 

-2.62 

0.61 

-0.51 

0.49 

292 

-1.34 

3.59 

0.12 

0.57 

345 

-1.37 

3.79 

0.67 

0.78 

267 

Level I - Upper 

-1.73 

1.02 

-0.3 

0.47 

592 

-1.57 

2.17 

0.08 

0.55 

656 

-1.62 

2.47 

0.39 

0.76 

579 

Level II - Bottom 

-1.4 

1.22 

-0.15 

0.47 

269 

-1.5 

1.51 

0.09 

0.55 

298 

-1.84 

2.34 

0.27 

0.69 

264 

Level II - Middle 

-1.74 

1.38 

-0.05 

0.48 

624 

-1.77 

2.32 

0.09 

0.54 

698 

-2.66 

3.58 

0.14 

0.77 

615 

Level II - Upper 

-1.65 

1.47 

-0.04 

0.49 

444 

-2.3 

1.65 

0.14 

0.5 

471 

-2.77 

2.7 

0.19 

0.74 

436 

Level III - Bottom 

-1.82 

1.25 

-0.04 

0.47 

260 

-1.81 

1.26 

0.23 

0.48 

272 

-2.22 

2.28 

0.28 

0.69 

253 

Level III - Middle 

-1.27 

1.54 

0.07 

0.55 

135 

-3.2 

1.31 

0.15 

0.58 

142 

-3.86 

1.99 

0.1 

0.87 

132 

Level III - Upper 

-0.69 

1.01 

0.03 

0.57 

11 

-0.76 

0.79 

0.02 

0.56 

10 

-1.56 

1.48 

0.09 

0.95 

9 

Level IV - All 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

6 

Total 

-2.62 

1.81 

-0.16 

0.51 

2,634 

-3.2 

3.59 

0.11 

0.54 

2,908 

-3.86 

3.79 

0.29 

0.77 

2,561 


Note: 2,908 students had academic change scores in 2009-10 and achievement level scores in 2008-09. 2,634 of these same students had academic change and 
achievement level scores in 2008-09. Furthermore, 2,561 of the 3,640 students served in 2009-10 had academic change scores in 2007-08 and 2008-09 and 
achievement levels in 2008-09. Min reflects minimum of lowest scores for students with Max reflecting the highest scores. SD is the standard deviation. 

Table 18 

Academic Change Score Statistics for 2009-10 Cohort of Participants by Program Completion 


Program Completion 

S 

pring 2009 

S 

pring 2010 

Difference 


Min 

Max 

Mean 

SD 

n 

Min 

Max 

Mean 

SD 

n 

Min 

Max 

Mean 

SD 

n 

Completed 100% 

-1.99 

1.38 

-0.18 

0.48 

851 

-1.77 

3.59 

0.2 

0.53 

931 

-2.66 

3.79 

0.39 

0.73 

832 

Completed 80-99% 

-1.82 

1.36 

-0.21 

0.51 

360 

-1.57 

1.86 

0.15 

0.53 

391 

-1.94 

2.55 

0.35 

0.76 

353 

Completed 80% or More 

-1.99 

1.38 

-0.19 

0.49 

1,211 

-1.77 

3.59 

0.18 

0.53 

1,322 

-2.66 

3.79 

0.38 

0.74 

1,185 

Completed Less Than 80% 

-2.62 

1.81 

-0.13 

0.53 

1,423 

-3.2 

2.12 

0.06 

0.54 

1,586 

-3.86 

3.58 

0.21 

0.79 

1,376 

All 

-2.62 

1.81 

- 0.16 

0.51 

2,634 

-3.2 

3.59 

0.11 

0.54 

2,908 

-3.86 

3.79 

0.29 

0.77 

2,561 


Note: 2,908 students had academic change scores in 2009-10 and achievement level scores in 2008-09. 2,634 of these same students had academic change scores in 2008-09. 
Furthermore, 2,561 of the 3,640 students served in 2009-10 had academic change scores in 2007-08 and 2008-09 and achievement levels in 2008-09. 
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Table 19 

Academic Change Score Statistics for 2009-10 Cohort of Participants by AoR Pre Level 


AoR Pre 

S 

pring 2009 

S 

pring 2010 

Difference 

Min 

Max 

Mean 

SD 

n 

Min 

Max 

Mean 

SD 

n 

Min 

Max 

Mean 

SD 

n 

Below Basic 

-1.99 

1.56 

-0.14 

0.51 

1,872 

-2.3 

3.59 

0.08 

0.53 

2,080 

-2.88 

3.79 

0.25 

0.77 

1,809 

Basic 

-1.71 

1.81 

-0.17 

0.5 

529 

-3.2 

1.81 

0.16 

0.54 

569 

-3.86 

2.34 

0.35 

0.74 

521 

Proficient 

-1.74 

1.25 

-0.21 

0.52 

114 

-1.08 

2.32 

0.27 

0.56 

124 

-1.84 

3.58 

0.47 

0.78 

114 

Advanced 

-1.04 

1.11 

-0.22 

0.51 

43 

-1.18 

2.17 

0.21 

0.63 

50 

-1.2 

1.56 

0.36 

0.76 

43 

Graduate 

-2.62 

1.18 

-0.26 

0.6 

76 

-1.32 

1.96 

0.24 

0.57 

85 

-1.56 

3.3 

0.49 

0.85 

74 

Total 

-2.62 

1.81 

- 0.16 

0.51 

2,634 

-3.2 

3.59 

0.11 

0.54 

2,908 

-3.86 

3.79 

0.29 

0.77 

2,561 


Note: 2,908 students had academic change scores in 2009-10 and achievement level scores in 2008-09. 2,634 of these same students had academic change scores in 
2008-09. Furthermore, 2,561 of the 3,639 students that took a Pre Test served in 2009-10 had academic change scores in 2007-08 and 2008-09. Overall pre-, 
post, and improvements (differences) are shaded. Differences above the overall average are italicized. 


Table 20 

Academic Change Score Statistics for 2009-10 Cohort of Participants by Post Level 


AoR Post 

Spring 2009 

Spring 2010 

Difference 

Min 

Max 

Mean 

SD 

n 

Min 

Max 

Mean 

Std 

n 

Min 

Max 

Mean 

SD 

n 

No Post Test Taken 

-1.73 

1.54 

-0.11 

0.52 

367 

-3.2 

1.37 

0.02 

0.56 

391 

-3.86 

2.34 

0.16 

0.8 

346 

Below Basic 

-1.99 

1.47 

-0.16 

0.52 

976 

-1.81 

2.12 

0.03 

0.53 

1,107 

-2.77 

2.7 

0.21 

0.77 

938 

Basic 

-2.62 

1.38 

-0.15 

0.5 

556 

-1.77 

2.24 

0.16 

0.53 

602 

-2.66 

3.26 

0.33 

0.73 

550 

Proficient 

-1.74 

1.36 

-0.18 

0.5 

241 

-1.08 

3.59 

0.21 

0.55 

262 

-1.84 

3.79 

0.4 

0.79 

239 

Advanced 

-1.85 

0.94 

-0.26 

0.48 

164 

-1.21 

2.32 

0.25 

0.53 

187 

-1.69 

2.35 

0.48 

0.71 

164 

Graduate 

-1.91 

1.81 

-0.13 

0.53 

330 

-1.72 

1.96 

0.25 

0.52 

359 

-1.77 

3.3 

0.38 

0.81 

324 

Total 

-2.62 

1.81 

- 0.16 

0.51 

2,634 

-3.2 

3.59 

0.11 

0.54 

2,908 

-3.86 

3.79 

0.29 

0.77 

2,561 


Note: 2,908 students had academic change scores in 2009-10 and achievement level scores in 2008-09. 2,634 of these same students had academic change scores in 2008- 
09. Furthermore, 2,561 of the students served in 2009-10 that took a Post Test had academic change scores in 2007-08 and 2008-09 
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Teachers’ Views 

Teacher survey results on the helpfulness of AoR for students based on their initial EOG Level 
scores corresponded well with the actual results students obtained at both extremes; teachers 
were likely to see Level I students and low Level II students as likely to benefit, and students 
with mid-range Level III and higher scores as unlikely to be helped (see Appendix 3a for 
details). These perceptions match the actual results based on academic growth. The few 
teachers interviewed indicated Level I students were not likely to benefit. 

However, the actual results suggested fewer students who initially scored in the mid to upper 
range of Level II benefited, while teachers considered AoR somewhat helpful or very helpful for 
these students. Also, low Level III students showed relatively strong results with AoR, while the 
teachers had very mixed views on the helpfulness of AoR with this group (Appendix 3, Item 17). 

A separate question asked teachers which types of students were likely to be helped with AoR. 

• The majority of respondents believed a variety of groups were likely to be helped by AoR. 
When we examined responses of “Very Helpful” and “Somewhat Helpful” combined, 
students who worked well independently (75%), students with mid-range scores on screening 
instruments (72%), LEP students (68%), and students with low scores on the screening 
instruments (67%) were considered most likely to benefit. Next were those with failing 
grades in English/Language Arts the prior year (65%) and OCS students (58%). 

• Teachers were most likely to respond that AoR was not likely to be helpful (“slightly 
helpful” or “not helpful” for students with attendance or behavior issues (72% and 71%, 
respectively). 

Questions: What Costs and Benefits Were Anticipated and Realized with AoR? 

Cost Analysis 

Cost analysis techniques help determine the value of an intervention and the benefits derived; 
cost analysis can take several forms. If we know the resources needed to implement a program, 
and the benefits expected can be quantified, we can conduct a cost analysis as one piece of 
evidence about the value of the program. These analyses generally take into account the number 
of persons impacted, the amount of time over which the benefit may be assumed to exist, and the 
opportunity costs associated with the selection of intervention. Cost analysis results should be 
taken into consideration along with other evaluation information in decision making (Levin & 
McEwan, 2001). 

Cost effectiveness is the cost analysis technique most appropriate to AoR (Levin & McEwan, 
2001). Cost effectiveness is designed to compare the costs and effects of two or more 
alternatives with similar objectives. It focuses on which alternative yields a given level of 
effectiveness for the lowest cost (or the highest level of effectiveness for a given cost). In the 
case of AoR, we chose to treat the varying pretest levels on EOG/EOC as the alternatives for 
treatment. An alternative program with the same goals is not available for comparison, and AoR 
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assessments are not given to all students. Thus, we were examining which type of students were 
the best candidates for this alternative based on their pretest scores. Data was not available on 
another literacy alternative for students with the same type of foundational reading skill deficits 
and characteristics. Cost-effectiveness models can be used when costs can be expressed in 
monetary terms and outcomes can be expressed in standard units of effectiveness. One measure 
of effectiveness is chosen, and it should reflect the main objective of the alternatives. 

Costs and Resources 

AoR has been a considerable investment for WCPSS over time ($1.8 million). Continuing costs 
are considerably lower than initial costs. The monetary costs of AoR over time in WCPSS are 
shown in Table 21. The software licenses are one-time expenditures per school for the life of the 
program. Technology support is an annual cost that provides updates to the software and 
technical support for software issues. Professional development costs and substitute costs vary 
over time. WCPSS is exploring ways to reduce training costs for AoR now that the program is 
in place in most schools and many teachers have two years of experience.. 


Table 21 

AoR 2009-10 Expenditures 


Expense Category 

2008-09 and Prior 

2009-10 

Total 

Software Licenses — one-time cost 

$769,897 

$258,840 

$1,028,737 

Technology Support — annual cost 

$30,216 

$50,000 

$80,216 

Professional Development from TE21 — annual 
adjustable cost 

$302,000 

$378,000 

$680,000 

Substitute Teachers for Professional Development — 
annual adjustable cost 

$7,000 

$7,000 

$14,000 

Total Expenditure 

$1,109,113 

$693,840 

$1,802,953 


Data Source: Budget information from program staff. 


Other costs which must be considered are school resources in terms of equipment, space, and 
staff time. We chose not to monetize these costs, because most are opportunity costs, meaning 
that choosing to implement the AoR program eliminated the opportunity to implement another 
option (but may not have cost any more to do). It should be recognized that offering smaller 
classes for AoR could lead to larger sections of other classes — particularly in English. 






Schools utilized teachers within their regular allotments to staff 
the program. Thus, other electives may not have been offered 
as a result (an opportunity cost), or one English I section may 
have been tailored to utilize AoR and be consequently smaller 
than other classes (making the others slightly larger). The 
counselor or administrator in charge of scheduling must also do 
purposeful scheduling and check the most recent test results 
and other information to correctly place students into AoR. 

Students spent time in AoR that might have been spent in other 
remedial or non-remedial pursuits — again, this is an 


r 




Opportunity Cost: 

Using resources in one 
way which takes away 
the opportunity to use 
them in another way. 
Resources are generally 
monetary, student time, 
or teacher time in 
education. 

< > 
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opportunity cost. It is important to assign students who will benefit from the program. 
Otherwise, students lose time that could be spent on more appropriate learning opportunities. 

• Central staff coordinated the effort and the contracts, and therefore incurred an opportunity 
cost associated with the program. This cost will increase if central staff assumes more of the 
training and support functions for AoR. 

Benefits 

The primary benefit seen at this point is an increase in the percentage of students able to reach 
their growth targets (AC scores) on the EOG/EOC. Increases in foundational reading skills are 
assumed to be the reason, although program completion and assessment data provide weak 
evidence to support that view. The hope is that improved reading skills will lead to greater 
comprehension of class materials, which in turn will lead to greater success in terms of grades, 
staying in school, and graduating. 

Test results showed there was a 25% increase in the percentage of students meeting or exceeding 
growth targets in 2009-10 as compared to 2008-09, the year before participation in the program. 
AoR improved from being less likely to as likely to show one year of growth after a year of 
instruction. The percentages reflected in Table 22 are reflective of only students who had scores 
in both years to demonstrate the program’s potential improvement. 

Table 22 

Percentages of Students Meeting or Exceeding Growth Targets on 2009-10 EOG/EOC 


Year 

Number 

Percent 

2008-09 

952 

37.2% 

2009-10 

1,582 

61 .8% 

Difference 

t 630 

T 24.6% 


Note: Alternative assessments do not yield AC scores and are therefore not included. 
n= 2,561 (students with scores in both years) 


All prior EOG achievement levels had more students meeting or exceeding their growth target 
after service than before. Table 23 shows that students below the targeted range of Level II - 
High or Level III - Low prior to service seemed to exceed growth targets at a greater rate than 
was true for both those in the targeted range and above. Moreover, students scoring Level I 
showed the greatest improvement rate in exceeding their growth target. 
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Table 23 

Percentages of Students Meeting or Exceeding Growth Targets on 2009-10 EOG/EOC 

by 2008-09 EOG/EOC Achievement Level 


Level 

2008-09 

2009-10 

n 

Difference 

Level I - Middle 

1 1 .6% 

59.9% 

267 

t 48.3% 

Level I - High 

25.0% 

59.4% 

579 

t 34.4% 

Level II - Low 

36.4% 

61 .0% 

264 

t 24.6% 

Level II - Middle 

45.4% 

57.1% 

615 

t 11.7% 

Level II - High 

47.2% 

63.8% 

436 

t 16.6% 

Level III - Low 

46.6% 

74.3% 

253 

t 27.7% 

Level III - Middle 

52.3% 

67.4% 

132 

(15.1% 

Level III - High 

k 

•k 

9 

* 

Level IV - All 

k 

k 

6 

* 

All 

37.2% 

61.8% 

2,561 

(24.6% 


Note: Alternative assessments do not yield AC scores and are therefore not included. Also, the 
rates are determined for students with AC scores in both years. 

* = Not shown due to sample size (10 or fewer students). 

Level I - Low has been omitted because no students fit the criteria. 


Cost Effectiveness Ratios 


In our cost analysis, we will focus on 2009-10, when 3,640 students were served. The one-time 
cost of licenses will be included as an amortized cost over a ten-year period. Some software 
would become out-of-date much faster, but the annual updates should maintain the value of the 
software tool. To summarize the direct costs for 2009-10, we included software licenses 
($1,028,737/10 years) + technology support at $50,000 (a district rate), professional development 
from TE21 ($378,000) + substitute costs ($7,000). These costs 
total $537,873.70. Thus, the cost per student in 2009-10 was 
$147.77. The cost per school was $9,779, which is important to 
recognize since the cost per student actually varies by school 
(since the number of students served varies). 


In order to determine whether students showed statistical 
improvement after participation in the AoR, a repeated measures 
t-test was conducted on students’ growth scores from the year 
they participated in the program (2009-10) as compared to the 
prior year (2008-09). Results showed that students 
demonstrated significant improvement (t= 1 8.77, df=2,560, 
pc.OOOl) in growth after participation than before; this would be 
highly unlikely to happen by chance. The expectation for 
students statewide is that they will have an academic change 
score of 0 or better. These students moved from an average 
growth of less than 0 to an average annual growth of more than 
0. After statistical significance was determined, an effect size 
was calculated using Cohen’s d formula to determine a different, 
more standardized measure of student improvement. The results 


Effect size shows how much 
difference a program/ 
intervention makes. It is the 
mean of the differences of 
the two years ’ growth 
scores divided by the 
standard deviation of the 
differences — change is 
expressed in terms of 
portions of a standard 
deviation. 

AoR results for the 2009-10 
cohort yielded an effect size 
of .38, which is considered 
moderate and educationally 
significant. 
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showed that the program had a moderate effect (d=0.38) on students’ growth scores. Most 
statisticians would consider this an important educational effect. 

Taking the next step, we examined the total cost benefit by pretest EOG category from AoR 
participation. As can be seen in Table 24, the price per student by group increases based on the 
proportion of students in the group who were successful in exceeding their target (defined as an 
academic change score greater than 0). Based on the 2009-10 participants with growth scores, 
the cost per successful student (based on those with academic change scores equal to or above 0) 
was $245.69; those with costs below this average included students who scored high in Level II 
and in the low to mid-range of Level III. However, the number of students who improved was 
far greater in the high Level I and middle Level II range, with a slightly higher cost per 
successful student ranging from $253.08 to $262.45. 

Table 24 

Cost Per Student by Prior EOG Level Scores 


Level Score 

Exceeded 

Target 

Available 

Growth 

Scores 

Cost per 
Student 

Total 

Cost 

Cost Per 
Successful 
Student 

Level I-Low 

* 

* 

- 

- 

- 

Level I-Mid 

192 

345 

$147.77 

$50,981 

$265.52 

Level I-High 

383 

656 

$147.77 

$96,937 

$253.10 

Level II-Low 

174 

298 

$147.77 

$44,036 

$253.08 

Level II-Mid 

393 

698 

$147.77 

$103,143 

$262.45 

Level II-High 

294 

471 

$147.77 

$69,600 

$236.73 

Level Ill-Low 

201 

272 

$147.77 

$40,193 

$199.97 

Level Ill-Mid 

95 

142 

$147.77 

$20,983 

$220.88 

Level Hi-High 

5 

10 

$147.77 

$1,478 

$295.54 

Level IV- All 

* 

* 

- 

- 

- 

Total 

1,735 

2908 

$147.77 

$429,715 

$245.69 


Note: Calculations were not run for groups of students who had fewer than 10 growth scores 
available — see asterisk (*). 
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CONCLUSION 

Overall, the AoR program in 2009-10 in WCPSS appeared to attain positive and promising 
results on EOG reading growth. The number of students served increased substantially, from 
1,358 in 2008-09 to 3,640 in 2009-10 as more schools were added. Program completion rates 
improved across the two years, but were still lower than would be optimal. The program 
appeared to be somewhat more helpful for middle school than high school students, both in terms 
of program completion and EOG/EOC reading growth. 

Students Served: School staff used a variety of criteria to identify students for service in AoR, 
despite district guidelines. Fewer students scored in the primary EOG recommended range (of 
high Level IIs and low Level Ills) than scored below it (the secondary target range). 

Implementation: AoR was most commonly utilized in remedial electives in 2009-10 (44%), 
with Curriculum Assistance and regular English I classes the next most common (at about 15% 
each). Overall, the teachers appeared to be adequately trained and the program appeared to be 
well implemented. Teachers liked and disliked aspects of the program’s use of technology, the 
content, the pacing, the motivational aspects, and the data provided. Teachers liked the focus on 
basic skills, the self-pacing, the motivators, the available data feedback, and the opportunity to 
work one-on-one with students. Commonly reported dislikes generally related to ways to refine 
the program, including the slow pacing, unrealistic standards for some exercises, students’ 
frustration and boredom with the materials, voice quality of the narrator, and immature 
characters in the materials. 

Outcomes: Program completion rates were only 26% in 2008-09 and 41% in 2009-10. Thus, the 
objective that all students would complete at least 80% of the program was not met. AoR met 
the EOG/EOC objective that students would improve in their reading comprehension as 
measured by Reading or English. Based on the 2009-10 cohort, the percentage of students 
reaching their growth target increased from 37.5% before service to 60.2% after service. 

Average academic change scores increased from below to above the target. Students appeared to 
benefit, on the average, whether they finished at least 80% of their program or not (with those 
completing the program slightly more likely to show growth). 

Overall, AoR did appear to meet the need of addressing many participants’ foundational reading 
skills. However, improvements are possible. 
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DISCUSSION 

Need: Clearly, some middle and high school students lack important foundational reading skills. 
Improving these students’ basic skills in reading is an appropriate goal given that it should 
enable them to improve their reading comprehension. It appears AoR can be part of the answer 
for WCPSS, especially if we address the issues raised in the recommendations below. 

Students Served: It is not uncommon for E&R to find the students served are not necessarily 
those who were described as the primary target for the program or service. Why is this? One set 
of reasons appears to be logistical. Placement is a multi-step process. EOG scores are listed as 
criteria, but referring teachers must rely on year-old scores to meet scheduling timelines. In 
addition, teachers must have access to assessment, classroom performance, and other data which 
they consider valid, or at least more valid, than the priority targets provided by central office. 

The scheduler or teacher at the receiving school should review the current test scores and 
additional data available on student needs. With the large number of students at each school, 
purposeful scheduling can be time-consuming, and it may or may not always happen. Receiving 
teachers have varying amounts of control over whom they serve, and criteria being used vary 
accordingly. Additionally, the communication of the priorities may be unclear, or not 
communicated, to all appropriate staff. Some staff may believe that students in the designated 
ranges will not benefit as much as others, and therefore adjust the criteria for selection. Finally, 
student placement into the appropriate program may be constrained by staff available and course 
schedules. In the case of AoR, all of these factors appear to be at work. 

Placement and Progress Categories: Loh (2009), who conducted research on the AoR product 
for AutoSkill, cites NAEP results indicating that 29% of students have reading skills two years or 
more below grade level upon entering high school. NAEP has high standards for basic 
proficiency. AoR may be using similar high standards in their placement categories, since many 
WCPSS students scored Below Basic when initially placed, even if they scored proficient on the 
state’s EOG the previous spring. The fact that fewer than half reached “Graduate” status may 
also be reflective of these standards. Loh (2009) indicates that each category represents one 
grade level of improvement, so students who move up two levels are increasing by two grade 
levels in one semester. High standards are not a problem, as long as all staff members associated 
with the program understand this. The number of skills necessary to complete the program was 
actually reduced in 2009-10, but completion was still an issue in WCPSS. Staff may want to 
explore whether more students should participate two semesters or whether an increase of two 
categories should be the target (which would still be challenging but more realistic). 

Data collection: Another common issue in conducting program evaluations is having easy 
access to necessary data. In the case of AoR, several issues increased the time necessary to 
complete this evaluation. Student identification numbers of participants are critical in matching 
data across files for evaluations. Use of the AoR software tool is not designated in course codes 
(one way we can identify participants and IDs), and the files internal to the program did not 
include WCPSS student identification numbers. In addition, costs over time could not be easily 
located by program staff, as well as the rollout schedule of the program by school over time. In 
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addition, coaches’ ratings of implementation were not available in a usable electronic format and 
had to be re-keyed for analysis. All of these data were important to this evaluation. 

As part of our response to the Curriculum Management Audit (CMA) findings, we are working 
more closely with program staff in the initial phases of program implementation so intended 
outcomes are clear and appropriate data are collected along the way (and kept in an accessible 
location). This will allow future evaluations to be completed more quickly. 


RECOMMENDATIONS 

AoR is addressing a real need in WCPSS and appears to be having a positive impact on EOG 
results related to literacy. Initial costs for licenses are already paid, except for a few existing 
schools, and nearly every school has trained teachers as well. Thus, continuing costs will be 
lower than the costs incurred by the district in the past. We recommend continuation of the 
program, with improvements in the areas that follow. 

Target Services to Student Needs More Strategically 

Program staff provided school personnel with informed opinions of the types of students likely to 
benefit from AoR. Some school staff followed those recommendations, while others used their 
own standards based on their experience, capacity, or other reasons. Now that AoR results are 
available, it should be possible to tailor service more tightly to student needs, and to tailor class 
structures and staffing more carefully to meet student needs. The eligibility criteria could be 
simpler, broader but more specific, more clearly communicated, and more consistently followed. 
This may involve sharing these results, discussing why teachers utilize current practices and 
what would be acceptable to them, and monitoring student selection more closely next year. 

To be more certain that students placed in AoR truly have foundational issues with reading skills, 
placement guidelines must be reviewed and revised. It will be important to divide the process 
into screening that takes place by the sending teacher, the school scheduler, and the receiving 
teacher. Ideally, the revision process will involve both school and central staff. Final decisions 
should be communicated to all staff involved in writing, as well as in other ways. Contracted 
trainers should also be clear on the criteria for service WCPSS is using to identify program 
participants; doing so should avoid conflicting communications with the schools. 

• The classroom teacher making the initial recommendation for possible AoR support should 
consider prior EOG scores, but also more current assessment and classroom performance 
data related to foundational reading skills. EOG scores are only one reflection of reading 
achievement, and actually reflect comprehension more than fundamental reading skills. 

EOG scores are also not available for the current year until after initial recommendations are 
due. Additional assessment data that are more sensitive to fundamental reading skills might 
include universal screening results with a Cloze paragraph assessment, and/or an oral reading 
fluency measure. Better guidance for teachers of ESL and SWD students is recommended. 
Some teachers seemed to assume that all ESL and SWD students would benefit; this 
perception was not supported by the data. English proficiency levels should be adequate to 
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understand the materials, and IEPs should reflect a need for developing foundational reading 
skills. 

• Schedulers should consistently consider the most recent EOG scores and other information 
from sending teachers before placing students in AoR. While some evidence suggests the 
recommended range of high Level IIs and low Level Ills are the ones most likely to benefit, 
most evidence suggests the range of prior EOG scores recommended for AoR can be 
broadened to include high Level I through low level III students (and some middle Level Is). 
Low Level Is or higher Level III or IV students are unlikely to benefit. Schedulers must also 
consider whether the class will be a homogeneous elective for reading support, or a 
classroom with a broader range of students such as English I. If a broader range of students 
is present, a paired teaching situation (perhaps with a regular and a Special Education 
teacher) may be necessary, with careful scheduling of content so basic instruction is not 
missed. A teacher must monitor progress as students work, and provide individual support to 
students as needed, and students should not miss core instruction in any subject. The broader 
range of students considered for the program may mean that more classes should be 
structured to include AoR instruction. 

• Once student schedules are set, the receiving teachers must carefully consider whether all of 
their students in their classroom actually need the program. If students do not all need AoR, 
appropriate adjustments should be made to students’ schedules or assignments (e.g., moving 
them to another class or having the student use other materials as soon as they complete the 
few skills not mastered). If students pretest as Graduate or Advanced, they would have 
already met the goal of the program, and there is little or no benefit to enrollment. About 5% 
of those served in 2009-10 scored Advanced or Graduate initially (149 students in 2009-10), 
which represents an opportunity cost since they could have been receiving instruction more 
appropriate for their learning needs. 

Make Improvements to the Program and Training to Increase Student Success 

Teachers raised some valid concerns about the program which, if addressed, could impact 
student success. Some would involve changes to the software program itself, while others could 
be handled internally within WCPSS. Central staff indicates that other concerns and comments 
made by teachers reflect a misunderstanding of the importance of automaticity to students’ future 
success and their role as coaches when students become frustrated; these concerns may be best 
addressed through additional communication and/or targeted training. 

WCPSS central staff should contact the vendor to see which concerns can be addressed on short- 
term and long-term timelines to improve the program. Some should be relatively easy 
adjustments, such as correcting the obvious glitches in the answer keys. Addressing issues like 
the narrator’s accent, the visuals used, or adding other features could be longer-term goals. 

Three issues stand out as most important for sustained improvement. 

• Lirst are the frustration, boredom, and pacing issues which were frequently mentioned in 
relation to AoR. Part of this relates to the requirement for meeting both accuracy and 
timing/automaticity standards to move forward in the program. A few teachers praised the 
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program for not allowing students to move on until they truly mastered the skills, but more 
teachers cited negative consequences, such as students giving up or not completing the 
program. Survey responses suggest some misunderstandings of AoR options and intent. 
Additional professional development for teachers may be needed on the foundational reading 
skills (particularly automaticity) and on how to better coach to students’ instructional needs 
when they get “stuck” in their lessons. Teachers may also not know all the options for 
adjusting instruction that are already available. If training does not make a positive impact, 
exploring ways to make other adjustments to AoR standards may be necessary. 

Improvements should increase program completion and success rates. 

• Second is the limited comprehension and lack of vocabulary material in the program. 

Adding more material in these areas would be advantageous to both WCPSS and the 
publisher. These are part of a balanced literacy approach, and comprehension skills are vital 
to student success. Other products may include these areas to a greater extent, but we have 
already invested a great deal in AoR, so it would be best to explore whether this could be 
added to the software before making any additional investments. Some teachers have 
identified and used materials to build comprehension once students complete AoR, but a 
uniform approach could be helpful. Students who complete their training stream early would 
then have a next step readily available. 

• Third, the goal of 100% completion of AoR should be discussed with staff, given that less 
than half of participants reached that standard. It may be that the goal could be an 
improvement of two categories or more (meaning all students would reach at least proficient 
status [at grade level] or above) or that more students should be recommended for a second 
semester of AoR. Improvement in reaching academic change growth targets was evident 
after AoR service, but was only slightly better for those completing the program than not. 
Still, 40% of students did not reach their academic change targets after service, and it may be 
that additional service could be helpful. 

Consider the Best Classroom Structure and Level for these Services 

AoR is used both within regular English/Language Arts classes and in a variety of elective 
settings. School-based and central services staff could benefit from discussing the various 
settings within which AoR is currently provided, as well as ways to optimize them. This could 
lead to recommendations for the most ideal settings to use, or to structure guidelines for making 
the most of the settings that cannot be changed. 

A second issue for discussion and consideration is whether the program should become more of 
a middle school resource over time. Program completion and EOG growth favored middle 
school students over high schools students, and it would be advantageous to have middle school 
students helped before entering high school. The issue of the degree of need at each school 
should also be addressed. It is conceivable that some elementary schools may actually need the 
program more than some high schools which have few low achievers. Transferring licenses 
from some low need schools to high need schools could lead to cost savings. 
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APPENDIX 1 

EVALUATION METHODOLOGY 
Program Staff and Vendor Interviews 

In the course of data collection, the Data Analyst and Senior Director for Program 
Accountability had several meetings with central staff contacts (the Director of Literacy and 
SWD Senior Administrator for Literacy, and one with two representatives of the vendor (TE21). 
Establishing the logic model was the first step, securing needed data from program staff was a 
second step, asking questions as they arose was a third step, and reviewing results was a fourth 
step. Some delays were encountered in the process of data collection, as well as some technical 
issues described in the appropriate sections. 

Teacher Survey 2009-10 

Questions were developed in collaboration with program staff after a variety of questions about 
student selection and program implementation were raised. E&R staff drafted the questions and 
then solicited feedback and suggestions from program staff. Teacher names were secured from 
teacher rosters and observation records, and 193 surveys were sent directly to the 2009-10 AoR 
teachers via email, with a link to a Zoomerang electronic survey. As it turned out, about 13 of 
the staff to whom email were distributed had either left the system or taught Academy of Math 
rather than Reading. Therefore, the final sample size was 180. In order to obtain the results in 
time for analysis and reporting, the survey window was limited to October 20 to November 3. 
With one reminder, the return rate was 63 teachers (35.0%). While this was lower than desired, 
it was fairly representative of the middle and high school distribution of students served, with 
roughly two-thirds being middle school teachers and one-third being high school teachers. 

Website and Articles 

The primary articles reviewed were those provided by staff as supporting the selection of AoR. 

A few other references on secondary reading research, secondary technology-based 
interventions, and AoR were also found via a web search. This study did not include a 
comprehensive literature review in the interest of time. 

AoR Participant Files 

AoR software tracks student data using the AoR software tool. Unfortunately, it does not record 
student identification numbers. These are essential to matching student results to other district 
files. In this case, use of AoR is often embedded as a tool within another course, so course 
numbers did not help to secure student identification numbers. 

For purposes of analyzing student outcomes, program staff provided lists of students’ names, 
program completion rates, pre-test scores and levels, post-test scores and levels, and numbers of 
levels gained/lost for each class and each school; these data were obtained from TE21. Initially, 
there were 4,178 students listed as participants, but 289 students were duplicate entries and were 
removed, resulting in 3,889 unique students. Because no student identification numbers were 
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provided, extra time was needed to write a SAS program to obtain each student’s NCWISE 
number using their first and last names, as well as their school. A fuzzy match search was 
employed to accommodate spelling errors; most but not all students could be found. Additional 
efforts were put in place to attempt to find district identification numbers for all students. 
Ultimately, 3,640 of 3,889 students (94%) could be located. Because there was no way of 
obtaining demographic and outcome data without identification numbers, the remaining 249 (just 
over 6% of the total sample) students were dropped from the sample. In some cases, these 
individuals were students who had left the district, changed names, or may have been a fictitious 
name used to demonstrate the software. 

EOG/EOC Test Scores 

EOG test scores in reading were secured for middle school students, and EOC scores in English I 
were secured for high school students via E&R student rosters. Missing test score data posed a 
challenge to any kind of longitudinal analysis. Specifically, two issues arose which reduced test 
scores available for students: a high concentration of students taking alternate assessments (for 
SWD and a few LEP students), and the focus of the outcome portion of the analysis on growth 
scores, each of which required test scores from two years. 

Growth scores were not computed for alternate assessments. Since these tests are quite different 
from one another, staff chose to focus on EOG/EOC exams for growth analyses and treat 
alternate assessments as their own category. Overall, only 2,921 students (or 80.2% of the total 
sample) had EOG/EOC results. The remaining 20% either took alternate assessments or were 
missing scores in at least one of the years. 

There were other circumstances where a student did not have a test score but was documented as 
participating in the program. First year LEP students are exempt from taking reading 
EOG/EOCs; other students may have come into the district late or left the district before testing; 
some students may have been absent on the day of testing. 

On the other hand, there were some cases where students who did not have prior EOG/EOC 
levels ended up having growth scores. These students were 8 th graders in 2007-08 and did not 
take the English I EOC in 2008-09, but instead took it in 2009-10. As such, these students did 
not have a level score for 2008-09, nor for the year prior to participation. Nevertheless, they did 
have growth score values as a result of taking the 2009-10 English I EOC — based on their 8 th 
grade reading EOG. 

Effect Size 

General descriptive statistics are shown below for 2008-09 and 2009-10 Academic Change (AC) 
scores for participants in the AoR in 2009-10. 


Year 

n 

Mean 

Standard Deviation 

Minimum 

Maximum 

2008-09 

2,634 

-0.16 

0.51 

-2.62 

1.81 

2009-10 

2,908 

0.11 

0.54 

-3.20 

3.59 

Difference 

2,561 

0.29 

0.77 

-3.86 

3.79 
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In order to evaluate the size of the treatment effect for AoR, Cohen’s d was computed, to 
determine the difference between two population means, which was then divided by the standard 
deviation from the data. Cohen’s d is a standardized measure of effect size for a repeated 
measures t-test: 


d 




Q.29 

0,77 


0.3S 


where is the mean and ov,e is the standard deviation of the difference in academic change 
scores from 2008-09 and 2009-10 (>< D = 0.29, - .77), The results ( d = 0.38) indicate that the 

program had a medium effect on student performance with respect to academic change from the 
year before and the end of the year of participation. 


Trainer Site Visit Reports 

AoR staff provided reports on implementation ratings to E&R in electronic files. While these 
files were electronic, they consisted of individual documents with color-coded ratings of 
performance for each teacher. Therefore, E&R staff had to manually key ratings from spring 
observations. Since we were interested in how teachers were rated by the end of the year, we 
started with the last observation of the year. If that was not available for a teacher, we went back 
to the prior observation to pick up additional teachers. 

In some cases, coaches were not able to directly observe implementation during a school visit for 
a specific teacher, yet were able to look at their students’ “time on task” percentages. In these 
cases, the teachers were rated solely on whether this value met the program’s objectives or not. 
Still, there were many teachers who were not observed or given implementation ratings. These 
teachers may have been at a school that was not receiving coaching support in 2009-10, or may 
have had section names that were unidentifiable. 


Cost Effectiveness Analysis 

Securing costs for the program was completed by the Director of Literacy, who had the data 
verified through the program’s vendor. Maintaining accurate costs over time was made more 
difficult by the fact that AoR does not have its own separate budget code, and the fact that a staff 
change occurred after AoR was launched. Staff will be encouraged to maintain accurate records 
regarding schools served, as well as future costs incurred, for evaluation purposes. 

The Cost-Effectiveness Analysis book by Levin and Me Ewan (2001) was used as a guide for 
calculating the costs and other benefits, as well as for the cost effectiveness model used in this 
analysis. 
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APPENDIX 2 

Coach Observation Form For Teachers 
TE21, Inc. Site Visit Summary 
FOR ADMINISTRATIVE USE ONLY 


Consultant: 

School Name 

Wake County Public Schools 

Date 

Teacher 

Grade and 
Product 

Computer 
Location and 
Time 

Observations of Teacher 

Implementation Review and 
Suggestions for Improvement 

Teacher Name 

Academy of 
Reading 

Grade Levels 

Lab or Class 
Number 

Time of 
class(es) 

Teacher effectively uses training monitor 
report. 

Teacher maintains an environment Q 

conducive to learning. 

Teacher quickly responds to students who Q 

are struggling. 

Teacher receptive to consultant’s suggestions i— « 

and ideas. 

Teacher actively using coaching strategies n 

that consultant has modeled. 

Teacher uses motivational strategies to _ 

promote student success. ^ 

Class maintains sufficient time on task in the _ 

program. 1— 1 


H Objective is being met by teacher D Needs improvement H Objective is currently not being met 


Academy of READING 
Teacher Training Summary 


Site Visit Codes: 

Teacher 

Initial 

Training 

Follow Up 
Training #1 

Follow Up 
Training #2 

Absent, Absent- 
Excused, No Usage, 
Leave, Track Out, 
Schedule Conflict 
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APPENDIX 3 

Zoomerang Survey Form and Responses 
Multiple Choice (3a) 

Academy of READING Teacher Survey 


Response Status: Completes 
Filter: No filter applied 
Nov 04, 2010 10:52 AM PST 


1. Please select your school level. 


Middle 

41 

65% 

High 

22 

35% 

Total 

63 

100% 


2. In which school years have you taught Academy of READING? (Mark all that apply). 


2010-11 

34 

54% 

2009-10 

61 

97% 

2008-09 

20 

32% 

Prior to 2008-09 

3 

5% 


3. In what type of class did you use Academy of READING in 2009-10? (Mark all that apply). 


Pull out from regular language arts or English 
class 

5 

8% 

Within regular language arts or English class 

9 

14% 

Remedial elective 

28 

44% 

Curriculum Assistance class 

10 

16% 

Other, please specify 

22 

35% 
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4. How many prior years of EOG/EOC scores did you use to select students for screening for possible service 
in Academy of READING in 2009-10? 


One year 

18 

29% 

Two years 

16 

26% 

1 did not have access to EOG/EOC scale 
scores. 

12 

19% 

1 had access to EOG/EOC scores but did not 
use them to select students. 

16 

26% 

Total 

62 

100% 


5. What other criteria did you use to select students for screening in 2009-10? (Mark all that apply). 


Blue Diamond results 

10 

16% 

CORE assessment results 

31 

50% 

Universal screening results 

10 

16% 

Classroom performance 

35 

56% 

Teacher or SST recommendation 

27 

44% 

None 

7 

11% 

Other, please specify 

20 

32% 


6. What EOG/EOC criteria did you use in 2009-10 to select students for screening for possible service in 
Academy of READING? (Mark all that apply). 

Level 1 

33 

52% 

Level II (bottom quarter) 

33 

52% 

Level II (mid-range) 

27 

43% 

Level II (top quarter) 

21 

33% 

Level III (bottom quarter) 

7 

11% 

Level III (mid-range) 

0 

0% 

level III (top quarter) 

0 

0% 

Level IV 

0 

0% 

1 did not use EOG/EOC scores to select 
students. 

24 

38% 
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7. Please explain how you determined the criteria that you used for screening. 


59 Responses 


8. Were you able to serve all of the students who appeared to have needs based on the screening in 2009-10? 


Yes 

34 

54% 

No 

29 

46% 

Total 

63 

100% 


9. Please explain why you were unable to serve all of the students who appeared to have needs. 


29 Respo nses 


10. How many days per week did identified students participate in Academy of READING in 2009-10? 


1 

0 

0% 

2 

7 

11% 

3 

27 

44% 

4 

7 

11% 

5 

21 

34% 

Total 

62 

100% 


1 1 . About how long were your Academy of READING sessions? 


25 minutes or less 

7 

11% 

30 minutes 

24 

39% 

45 minutes 

28 

45% 

50 minutes or more 

3 

5% 

Total 

62 

100% 
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12. What do you do if a student has completed his/her assigned 

1 work in Academy of READING early? 


Assign work for additional practice 

16 

26% 

Send the student back to regular class 

5 

8% 

Let the student do other work 

19 

31% 

Other, please specify 

22 

35% 

Total 

62 

100% 


13. What aspects of the program do you like best? 


5 9 Res p onses 


14. What aspects of the program do you like the least? 


59 Responses 


15. How could the program be improved? 


57 Responses 


16. What challenges have you faced while implementing Academy of READING? 


58 Responses 


17. Please indicate how helpful Academy of READING is for students with EOG/EOC scores in the following ranges in the 
prior year. (Respond for all ranges that apply to your Academy of READING students). 
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Top number is the count of respondents 

Very helpful 

Somewhat 

Slightly 

Not helpful 

selecting the option. Bottom % is percent of 
the total respondents selecting the option. 


helpful 

helpful 



18 

19 

15 

5 

Level 1 

32% 

33% 

26% 

9% 


20 

24 

11 

3 

Low Level II 

34% 

41% 

19% 

5% 

Mid-range Level II 

15 

26 

7 

5 

28% 

49% 

13% 

9% 

High Level II 

19 

16 

10 

6 

37% 

31% 

20% 

12% 


8 

8 

8 

12 

Low Level III 

22% 

22% 

22% 

33% 

Mid-range Level III 

5 

4 

1 

18 

18% 

14% 

4% 

64% 

High Level III 

3 

4 

2 

19 

11% 

14% 

7% 

68% 


2 

4 

2 

19 

Level IV 

7% 

15% 

7% 

70% 


18. Please indicate how helpful Academy of READING is for the following types of students. (Respond for all ranges that 
apply to your Academy of READING students). 


Top number is the count of respondents 
selecting the option. Bottom % is percent of 
the total respondents selecting the option. 

Very helpful 

Somewhat 

helpful 

Slightly 

helpful 

Not helpful 

Students with failing grades in 

10 

25 

13 

6 

English/Language Arts the prior year 

19% 

46% 

24% 

11% 

Students with the lowest scores on the 

18 

20 

15 

4 

screening instruments 

32% 

35% 

26% 

7% 

Students with mid-range scores on the 

10 

30 

13 

3 

screening instruments 

18% 

54% 

23% 

5% 


28 

13 

10 

4 

Students who work well independently 





51% 

24% 

18% 

7% 


9 

13 

12 

4 

Occupational Course of Study (OCS) students 


24% 

34% 

0^70 

1 1 % 


13 

21 

17 

3 

Students with Disabilities (SWD) 





24% 

39% 

31% 

6% 


18 

16 

11 

5 

Limited English Proficient (LEP) students 


36% 

32% 

22% 

10% 


3 

13 

22 

17 

Students who have behavior issues 






5% 

24% 

40% 

31% 


5 

10 

14 

26 

Students who have attendance issues 


9% 

18% 

25% 

47% 
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19. Please indicate how helpful you feel Academy of READING has been in addressing the following issues. 


Top number is the count of respondents 
selecting the option. Bottom % is percent of 
the total respondents selecting the option. 

Very helpful 

Somewhat 

helpful 

Slightly 

helpful 

Not helpful 


7 

12 

25 

18 

Comprehension 


11% 

19% 

40% 

29% 


39 

18 

5 

0 

Phonemic Awareness 


63% 

29% 

8% 

0% 


16 

26 

13 

7 

Fluency 


26% 

42% 

21% 

11% 


8 

17 

20 

17 

Vocabulary 


13% 

27% 

32% 

27% 


38 

16 

7 

1 

Phonics 


61% 

26% 

11% 

2% 


20. Did you receive training for Academy of READING? 



Yes 

61 

97% 

No 

2 

3% 

Total 

63 

100% 


21. Please indicate your level of agreement with the following statements. 




Top number is the count of respondents 
selecting the option. Bottom % is percent of 
the total respondents selecting the option. 

Strongly Agree 

Agree 

Disagree 

Strongly 

Disagree 

Training for Academy of READING helped me 

26 

31 

4 

0 

understand the program. 

43% 

51% 

7% 

0% 

1 was able to implement Academy of 
READING reasonably well with the training 1 

25 

32 

4 

0 

received. 

41% 

52% 

7% 

0% 

Academy of READING coaching improved my 

26 

29 

6 

0 

ability to use Academy o f READING. 

43% 

48% 

10% 

0% 

Academy of READING coaching was 
essential to my ability to implement the 

24 

25 

11 

1 

program effectively. 

39% 

41% 

18% 

2% 
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Appendix 3b 
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AoR Report: Illustrative Open-Ended Comments — 

What Teachers Liked Best and Least 

(Note: Comments are as written; only minor punctuation and spelling corrections were made) 


Liked Best 


Liked Least 


Weekly report allows students to monitor 
goals. 


All of the data available regarding student 
progress and performance. This is 
particularly helpful for creating IEPs and 
PEPs and addressing specific skill deficits 
that are impacting student's ability to read. 

Pretesting to assess level. 

I like the way the program keeps records 
and statistics on each student and on the 
class as a whole that are easily accessible to 
the teacher at any time. 

You can see how each child is doing in 
addition to the time on task. 


Placement tests. 

I am disappointed with some of the reporting tools. For 
example, when a student misses comprehension 
questions, I would like to see the entire passage. Also, 
there are responses that register as incorrect when the 
students are responding that appear differently when we 
review the errors the student has made. This happens 
too frequently and has been observed by multiple 
teachers. For example, the word will be “married”, the 
student selects the correct answer (visually and 
auditory) but the system marks the selected answer as 
incorrect. When we review the items missed, the item 
lists the word as “marred”. 


Since we have a "visual" through the puzzle 
that the students are creating on screen, we 
can see quickly students who are struggling 

without being tied to the computer. 

I liked the fact it’s multimedia; it’s 
interactive. 


Difficulty in passing certain tests. 


There are times when the program says the student has 
put in an incorrect answer when actually they have not. 


Students like using the computer. 

Kid friendly graphics. 

Measurable; able to show students and 
teachers progress immediately. 

Computerized. 


Teacher time —not for mistakes, but for too much time- 
means to me that they are really trying. 

The computer glitches were tough to manage. 

Computer network is unreliable, charts and graphs do 
not print, claims to work on reading comprehension but 
does not. 


The narrator’ s voice is difficult for the students to 
understand and hear the different pronunciations of the 
letter sounds. Kids from the south hear the accent or 
lack of accent and make mistakes on the different 
sounds. 


The little yellow blob appears juvenile to most students. 
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Students worked at their own pace and 
level. 

Independent; pinpoints areas of need. 

The program helps break down areas of 
weakness for each individual and they work 
to strengthen only those weak areas. 

Students liked the rewards; progress 
monitored. I like the individually 
diagnostic and prescriptive nature of the 
program. It places a student where they 
need to be and requires mastery of a needed 
skill before allowing them to progress to the 
next skill. 


Kids get stuck on the pacing. SO even though they may 
have achieved accuracy for 50 trials, they need to keep 
doing it until the pacing is the same. 

It can sometimes get tedious for them. The program's 
demand that they complete skills within a certain time 
limit tends to trap them in a skill that they really have a 
basic mastery of. Teenagers tend not to be very 
predictable and can get easily distracted. 

Hard to complete program unless you use it every day 
of the week day and continue with the program with 
results. 

ADHD students get frustrated easily and have trouble 
passing the tempo parts of the program. 


Flexibility — allows me to work with one 
student while others are on Academy. 

Allows teacher time to work one-on-one 
with students 

. . .everybody in the room can be working at 
a different level. 


Very repetitious— students got bored. 

Did not teach reading comprehension or vocabulary did 
not explain cultural differences, idioms. 

The initial pacing schedule. 

Repetitive tutorials. 


It’s boring as hell, modifying times and tasks is more 
ambiguous than it is productive, students who can read 
but do not have good grammar or comprehension test 
poorly and do not grow much, especially on EOGs, 
students become frustrated to the point of quitting 
because of the pacing that is required, despite adequate 
mastery and understanding (i.e. distractions, 
interruptions, ending class, or attitudes can all mess up a 
pace and cause students to have to start over on a skill). 
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I like the certificates. 

The students work hard to earn them. 
Students remained focused and are 
motivated to reach the next level. 

1 do think the program is good with 
motivated students. When success is seen 
by the student they want more. 

[Students] feel less embarrassed about 
doing foundational work. 

Motivating for middle school students. It 
provides positive feedback and allows for 
each student to work on their own levels. 

It’s really effective and empowering for 
ESL students and students that seemed to 
miss phonetic assembly for one reason or 
another. 

Repetition caused the students to focus in 
order to go to the next level. Giving up is 
not an option. 

I saw gains from students who were 
compensating in reading by skipping 
words. . .actually slow down and try to work 
out the words. 


Some students fought the program. 

The frustration of the students when they get stuck in a 
skill. . .. Student sometimes had to do same lesson 30 
times. 

Students need to have a certain degree of motivation 
and buy in to be successful. It is a little boring for them, 
but as the teacher I understand the importance of the 
skills and work to convey this to the students. 

It takes a lot of concentration on the students' part and it 
can be very frustrating to them. 

Program appeared to be on elementary level to student 
and therefore they were turned off immediately and did 
not want to do it. 

An extremely boring program. Too much repetition. 
Pretest is too long and students quit trying. 

Too much practice with some of the lessons made the 
students not want to continue. . .maybe condense some 
of it. 

It is a hard program if a student doesn’t speak English. 
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It’s great for improving their fluency. . . 

Comprehension, visual-sound match. 

I also like the focus on automaticity and 
fluency, since these are necessary for 
comprehension. 

The phonics and comprehension passages. 

The word building activities. 

Students learn the basic phonics and oral 
distinction of each letter or letter 
combination. 

I particularly like that the program 
emphasizes phonological awareness. 


I don’t feel that it helps students much with their 
comprehension. 

Lacked comprehension/vocabulary component that 
English Language Learners desperately need. 

The small amount of comprehension passages ...usually 
only see towards the end of the program. 

The reading passages are just checkups. The open 
syllables have short vowel sound. The program can be 
very frustrating for some students and easy for others. 

The pretest has many grammar and usage type 
questions, not comprehension. For example, the use of 
their of there and its or it’s. Students who read at grade 
level or just below are given similar lessons to those 
who read 2 or 3 years below. Vocabulary is missing. 

It stopped at 8 th grade; some wanted to go higher. 


Students getting easily frustrated when they could not 
pass a skill and there was not a different kind of activity 
to have them practice that skill. 


For the high school setting, the 
logos/illustrations/visuals are a little childish. This 
embarrasses some of the children. 


I was only comfortable if a student really needed this 
type of remedial help, many of my students were 
extremely frustrated by the program. 


62 






Academy of Reading Evaluation 


E&R Report No. 1023 


Appendix 3c 

AoR Report: Challenges and Suggestions 



Challenges 

Suggestions 

Data 


Shorter pretest. 

Vary the practice assessments. Include review 
games. Move from cartoon to actual video. 
Provide an option for the students to have the 
sound repeated. Increase the number of 
questions on the comprehension assessments. 

Have more of a focus on the end results of 
intervention for high school students 

The placement test could include activities to 
see if the student is hearing letters, and another 
person's voice should be used. 

Students should be able to view their own 
reports so they can self-monitor, reading 
comprehension needs improving, there should 
be an option for daily progress in addition to 
weekly progress OR the weekly report should 
have a daily column. 
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Computer access and dependability of the network I would like what the teacher can "see" as the 

are the largest hurdles. student responds to trials to be expanded. On the 

comprehension, I would like to "see" the stories 

We also need more time built into our schedule to without having to read over the student's 
work on the academy. shoulder, or to see the responses to 

auditory/visual exercises with 

Computers that are slow; students who grow tired of 

the program. Make more user friendly so that other teachers 

can do this as well. Fell to mostly Language 
For a while, the program would just not work Arts teachers, 

correctly on Thursdays... 

Lack of technical information, not enough computers 

Time, space, system crashing. 

It takes forever to put new students in your classes. 

At our school no one was interested in helping with 
the computer problems. It was hit or miss whether we 
could use them or not. It made for planning 
nightmares, students off task because of a change in 
the routine, and the data were not as valid as it should 
have been. It is frustrating. 

. . .program shut down during student trainings. 

I personally did not have many problems, but some 
of my colleagues had trouble with not enough 
computers or headphones, not being able to 
consistently schedule time in the computer lab, and 
difficulty taking time away from an academic class 
(usually English). 

Technology can trip us up a bit in the classroom (e.g. 
computers needing to be re-imaged; broken head 

phones). Nothing too major. 
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Some autistic students have a hard time getting the Make the fluency or pacing component within a 

pacing down. At times, it is difficult to move on. greater time such as 10-30 seconds as opposed 

Perhaps having the option to skip a lesson and come to 5. 
back to it the next session would have been helpful. 

Greater ease in moving to the next level. 

Students becoming frustrated when they have 

reached accuracy during an Auditory-Visual or I feel that if the program is going to be used 

Visual match skill, but not the pacing. with special education students with severe 

reading problems, the teacher should be able to 

I've had students who could not stand it, so we had a go in and modify the program as needed. In 
devil of a time getting them to buy in. addition a special needs student with serious 

reading problems needs more. 

Students get frustrated or bored with repeating 
certain tests. 

Keeping students motivated because some students 
became frustrated easily. 

The program is not for everyone. Some kids grow too 
frustrated with the program and start hitting keys 
instead of trying. I have several kids on it but some 
that do not need the fluency/ decoding piece and I 
challenge them on their levels with individualized 
work in targeted areas. 

I think when the program is broadly used through the 
school. Many teachers lose focus causing the students 
[to] lose focus. The program can cause loss of 
interest and boredom for students. The program 
needs to come from a central location with teachers 
that have interest in the program. It is not a filler 
program where students "play" on the computer. 

Most of them go through the motions, but I did not 
see any transfer to real-world reading. 

Intervention resistance. Some students just will not 
do it because it is reading remediation. . .high school 
students sometimes rebel at having a reading class. 

Students tend to listen to music or search the net 
unless the teacher is standing behind them. 

My students became very discouraged with the 
program. They had a hard time achieving mastery. I 
am not sure if AoR is the best program to be used 
with students who have severe reading problems. I 
supplemented AoR with Phonic games, tests, as well 
as reading comprehension stories that were on the 
level of my students. 
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It's EXTREMELY confusing and the slightest noise 

Get to the core readings faster. 


(including someone walking past) throws students off 
if they are doing the listening-visual match. 

More vocabulary and comprehension skills. 


Helping a student get through a lesson based on 

More comprehension. Have it mixed it instead 


pacing. I like to change the preset guidelines but 

of at the end of program. 


some kids struggle with the pacing and having 100 
plus trials is not uncommon for some students. 

More test analysis questions, more homophones. 


phonic sounds were unclear and difficult to 

more Greek/Latin roots. 


understand (Canadian woman) Program would freeze 
causing frustration to already frustrated students. 

Take away the demand that students complete 
skill sets within a specific time limit or within a 


Students who do not speak enough English to tell 

consistent time limit. 


them how to do it. 

Make the program a little more "mature" and 


Not enough training. 

multicultural. The characters that talk to them 
need to appear a little older/more mature. Some 


Being forced to do a program that you do not really 

of my students are past 18 years old. 


understand. 

I think the characters get on the students nerves 


Keeping students motivated when they are 

sometimes. For middle school a more mature 


struggling. 

character would be more appropriate. 



Change the pretest to be more on 
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If you could have the option of clicking on 
words and getting definitions in the 
comprehension passages. 



The program needs to be written with more 
mature graphics for HS students. 



More variety— teach reading strategies 



I wish there were more reading modules earlier 
in the program. 



Narrator for each region or state with common 
accent for each. 



They could provide preplanned supplemental 
interventions for when a student requires 
"teacher time." 



Have all lessons so that the student can have a 
sound repeated. 



Having a different kind of activity for a student 
to do when they were having a lot of difficulty 
with a skill. 
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One of me, and many students. I'd like more time to Change it to a direct instruction program, with a 
work with each student individually. teacher for four weeks during the summer. Kids 

who cannot read by the time they get to high 
Too many kids without enough support and I had to school get further and further behind in their 
implement additional work for those that completed classes - taking them away for even a short 
the program when I was already overwhelmed by too amount of time puts them at greater risk for 
many students at one time. failure. 

More students NEED to participate in this program. The program should be given to students that it 

but there is only room for so many students within is designed for. -The program is NOT for 

the class. students who do not speak a lot of English and 

they should not be forced to do it. -Inform 
administrators concerning who should and 
should not do this program. -Someone should 
explain exactly how this program has helped 
students and what are the study results and on 
whom. 
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